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Abstract 


Defence  Research  and  Development  Canada  (DRDC)  Toronto  is  in  the  process  of  developing  team 
research  scenarios  aimed  at  supporting  the  Canadian  Forces  (CF)  future  integrated  operations,  and 
interoperability  with  allies,  other  government  departments  (OGDs)  and  non-government 
organizations  (NGOs).  This  work  falls  within  a  4-year  Applied  Research  Project  (ARP)  to  include  a 
literature  review  of  relevant  team  literature,  the  creation  of  a  platform  for  conducting  experiments 
on  teams,  the  running  of  team  experiments  using  a  scenario  involving  one  or  more  Human  Systems 
Integration  (HSI)  intervention(s),  the  development  of  a  computational  model  of  team  performance, 
and  some  preliminary  validation  of  this  model.  Previous  reports  (Sartori,  Waldherr  and  Adams, 
2006;  Go,  Bos  and  Lamoureux,  2006)  have  reported  the  outcomes  of  exhaustive  literature  reviews 
on  team  research  and  team  research  platforms  respectively. 

This  report  describes  the  outcomes  of  two  parallel  streams  of  work.  The  first  stream  was  the 
development  of  three  team  experimental  scenarios,  in  a  domestic  operational  context,  appropriate  for 
studying  the  targeted  teamwork  factors  (i.e.  teams-of-teams,  joint,  interagency,  distributed 
environment).  This  was  done  by  identifying  and  reviewing  scenarios  used  previously  in  team 
research,  leveraging  concepts  important  to  team  research  scenarios  identified  by  the  literature 
review,  and  incorporating  knowledge  of  future  CF  requirements  in  new,  composite  team  research 
scenarios.  The  second  objective  of  this  report  was  to  evaluate  a  variety  of  computational  modelling 
applications  for  their  adequacy  in  modelling  the  targeted  teams  in  the  targeted  scenarios,  and  to 
recommend  one  application  as  the  most  suitable. 

This  report  provides  detail  regarding  the  different  scenarios  and  computational  models  evaluated, 
and  provides  direction  for  the  further  development  of  scenarios  to  suit  the  detailed  requirements  of 
the  ARP. 
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Resume 


Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  Toronto  elabore  actuellement  des 
scenarios  de  recherche  sur  les  equipes  afin  d’appuyer  les  futures  operations  integrees  des  Forces 
canadiennes  (FC)  et  I’interoperabilite  avec  les  allies,  avec  d’autres  ministeres  et  avec  des 
organisations  non  gouvernementales  (ONG).  Ces  travaux  s’inscrivent  dans  le  cadre  d’un  projet  de 
recherche  appliquee  (PRA)  d’une  duree  de  quatre  ans,  qui  comprendra  I’analyse  documentaire 
d’etudes  pertinentes  consacrees  aux  equipes,  la  creation  d’une  plate-forme  pour  mener  des 
experiences  sur  les  equipes,  la  realisation  d’experiences  a  partir  d’un  scenario  comportant  au  moins 
une  intervention  basee  sur  I’integration  des  systemes  humains  (ISH),  I’elaboration  d’un  modele 
informatique  de  rendement  d’equipe  et  certains  travaux  preliminaires  de  validation  de  ce  modele. 
Des  rapports  anterieurs  (Sartori,  Waldherr  et  Adams,  2006;  Go,  Bos  et  Lamoureux,  2006)  faisaient 
etat  des  resultats  d’ analyses  documentaires  detaillees  portant  sur  des  etudes  consacrees  aux  equipes 
ainsi  que  sur  les  plates-formes  d’etude. 

Le  present  rapport  decrit  les  resultats  de  deux  volets  de  recherche  paralleles.  Le  premier  volet  etait 
consacre  a  I’elaboration  de  trois  scenarios  experimentaux  dans  un  contexte  operationnel  national 
convenant  a  I’etude  de  facteurs  specifiques  du  travail  d’equipe  (c.-a-d.  des  equipes  d’equipes,  des 
equipes  interarmees,  interagences  ou  decentralisees).  Pour  ce  faire,  les  auteurs  ont  reuni  et  examine 
des  scenarios  ayant  servi  dans  le  cadre  d’etudes  anterieures  sur  les  equipes,  ils  ont  developpe 
d’importants  concepts  degages  par  I’analyse  documentaire  et  applicables  aux  scenarios  de  recherche 
sur  les  equipes,  ils  ont  integre  les  connaissances  sur  les  futurs  besoins  des  FC  en  matiere  de 
nouveaux  scenarios  pour  I’etude  d’equipes  composites.  Le  deuxieme  objectif  du  rapport  consistait  a 
evaluer  diverses  applications  de  modelisation  informatique  pour  verifier  leur  adaptation  a  la 
modelisation  d’equipes  dans  des  scenarios  cibles  et  recommander  I’application  la  plus  adaptee. 

Le  rapport  fournit  des  details  sur  les  divers  scenarios  et  modeles  informatiques  evalues  et  propose 
une  orientation  quant  a  I’elaboration  de  futurs  scenarios  qui  conviendront  aux  besoins  detailles  du 
projet  de  recherche  appliquee. 
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Executive  Summary 


Defence  Research  and  Development  Canada  (DRDC)  Toronto  is  in  the  process  of  developing  team 
research  scenarios  aimed  at  supporting  the  Canadian  Forces  (CF)  future  integrated  operations,  and 
interoperability  with  allies,  other  government  departments  (OGDs)  and  non-government 
organizations  (NGOs).  This  work  falls  within  a  4-year  Applied  Research  Project  (ARP)  to  include  a 
literature  review  of  relevant  team  literature,  the  creation  of  a  platform  for  conducting  experiments 
on  teams,  the  running  of  team  experiments  using  a  scenario  involving  one  or  more  Human  Systems 
Integration  (HSI)  intervention(s),  the  development  of  a  computational  model  of  team  performance, 
and  some  preliminary  validation  of  this  model.  Previous  reports  (Sartori,  Waldherr  and  Adams, 
2006;  Go,  Bos  and  Lamoureux,  2006)  have  reported  the  outcomes  of  exhaustive  literature  reviews 
on  team  research  and  team  research  platforms  respectively. 

This  report  describes  the  outcomes  of  two  parallel  streams  of  work.  The  first  stream  was  the 
development  of  three  team  experimental  scenarios,  in  a  domestic  operational  context,  appropriate  for 
studying  the  targeted  teamwork  factors  (i.e.  teams-of-teams,  joint,  interagency,  distributed 
environment).  The  second  stream  was  the  evaluation  of  a  variety  of  computational  modelling 
applications  for  their  adequacy  in  modelling  the  targeted  teams  in  the  targeted  scenarios,  and  to 
recommend  one  application  as  the  most  suitable. 

Both  streams  of  work  began  with  extensive  literature  reviews  to  identify  team  research  scenarios  and 
computational  models  of  interest  to  this  work.  The  search  resulted  in  a  high  number  of  possible 
scenarios,  so  scenarios  were  restricted  to  those  that  had  already  been  used  for  team  research,  as 
opposed  to  those  that  are  used  for  training. 

The  review  of  scenarios  was  conducted  against  a  number  of  criteria.  Based  on  the  number  of 
criteria  the  scenario  addressed,  it  was  deemed  highly  relevant,  relevant,  or  somewhat  relevant.  A 
total  of  37  scenarios  were  evaluated,  of  which  15  were  deemed  highly  relevant.  Based  on  the 
evaluation,  it  was  clear  what  criteria  were  consistently  addressed  in  team  research,  and  what  criteria 
have  been  seldom  addressed.  This  led  to  the  development  of  a  plan  to  create  new  scenarios  for  the 
purposes  of  team  research  at  DRDC.  This  plan  was  reviewed  by  the  Scientific  Authority  (SA)  and 
detailed  guidance  was  provided. 

Following  the  guidance  of  the  SA,  three  scenarios  for  team  research  in  the  CF  were  created:  a 
natural  disaster,  a  terrorist  threat,  and  an  influenza  pandemic.  These  scenarios  all  involve  a  three- 
person  CanadaCom  team,  a  three-person  Joint  Task  Force  team,  and  a  three-person  OGD  team 
(ranging  from  federal  to  provincial/local  level).  The  scenarios  all  have  the  capability  to  address  the 
team  research  factors  identified  by  Sartori  et  al  (2006)  as  well  as  being  structured  to  offer  a  medium 
level  of  fidelity  and  a  high  level  of  control.  A  template  is  also  provided  against  which  to  develop 
the  detailed  scenario  events  and  their  associated  measures  of  performance. 

In  common  with  the  scenario  development  work,  the  evaluation  of  the  computational  models 
proceeded  against  a  list  of  criteria.  A  total  of  26  modelling  applications  were  assessed  against  15 
criteria.  Given  the  requirements  of  the  team  research  ARP,  it  was  concluded  that  the  Integrated 
Performance  Modelling  Environment  (IPME)  was  the  most  appropriate  computational  modelling 
platform.  This  conclusion  included  the  belief  that  IPME  already  has  a  core  of  experienced  and 
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skilled  users  whieh  would  obviate  the  need  for  a  ‘new’  learning  eurve  associated  with  a  ‘new’ 
computational  modelling  application.  No  computational  modelling  application  offered  any  capability 
above-and-beyond  those  offered  by  IPME. 

Both  the  scenario  development  and  the  computational  model  review  have  the  potential  to  lead  to  real 
benefits  to  the  CF.  In  the  first  instance,  the  computational  model  can  quickly  and  effectively 
provide  insights  into  the  likely  impact  of  human-systems  integration  (HSI)  interventions,  as  well  as 
identifying  where  DRDC  should  focus  its  attention,  either  with  respect  to  HSI  or  with  respect  to 
team  research.  This  can  result  in  significant  time  and  financial  savings  with  a  greater  likelihood  of 
project  success.  Then,  having  embarked  upon  a  program  of  team  research,  the  insights  gained  will 
be  helpful  to  the  CF  when  structuring  teams,  providing  tools  to  enhance  team  performance,  and 
understanding  how  to  overcome  team  dysfunction  (especially  in  stressful  situations). 

This  work  was  performed  under  contract  W771 1-04791 1//001/TOR,  call  up  number  7911-05.  The 
SA  for  this  work  was  Dr  Renee  Chow. 
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Sommaire 


Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  Toronto  elabore  actuellement  des 
scenarios  de  recherche  sur  les  equipes  afin  d’appuyer  les  futures  operations  integrees  des  Forces 
canadiennes  (FC)  et  I’interoperabilite  avec  les  allies,  avec  d’autres  ministeres  et  avec  des 
organisations  non  gouvernementales  (ONG).  Ces  travaux  s’inscrivent  dans  le  cadre  d’un  projet  de 
recherche  appliquee  (PRA)  d’une  duree  de  quatre  ans,  qui  comprendra  I’analyse  documentaire 
d’etudes  pertinentes  consacrees  aux  equipes,  la  creation  d’une  plate-forme  pour  mener  des 
experiences  sur  les  equipes,  la  realisation  d’experiences  a  partir  d’un  scenario  comportant  au  moins 
une  intervention  basee  sur  I’integration  des  systemes  humains  (ISH),  I’elaboration  d’un  modele 
informatique  de  rendement  d’equipe  et  certains  travaux  preliminaires  de  validation  de  ce  modele. 
Des  rapports  anterieurs  (Sartori,  Waldherr  et  Adams,  2006;  Go,  Bos  et  Lamoureux,  2006)  faisaient 
etat  des  resultats  d’ analyses  documentaires  detaillees  portant  sur  des  etudes  consacrees  aux  equipes 
ainsi  que  sur  les  plates-formes  d’etude. 

Le  present  rapport  decrit  les  resultats  de  deux  volets  de  recherche  paralleles.  Le  premier  volet  etait 
consacre  a  I’elaboration  de  trois  scenarios  experimentaux  dans  un  contexte  operationnel  national 
convenant  a  I’etude  de  facteurs  specifiques  du  travail  d’equipe  (c.-a-d.  des  equipes  d’equipes,  des 
equipes  interarmees,  interagences  ou  decentralisees).  Le  deuxieme  objectif  du  rapport  consistait  a 
evaluer  diverses  applications  de  modelisation  informatique  pour  verifier  leur  adaptation  a  la 
modelisation  d’equipes  dans  des  scenarios  cibles  et  recommander  I’application  la  plus  adaptee. 

Les  deux  volets  des  travaux  ont  commence  par  des  analyses  documentaires  detaillees,  pour  reperer 
des  scenarios  de  recherche  sur  les  equipes  et  des  modeles  informatiques  pertinents.  Ce  travail  a 
degage  un  grand  nombre  de  scenarios  possibles,  et  il  a  etc  decide  de  s’en  tenir  aux  scenarios  qui 
avaient  deja  ete  utilises  dans  le  cadre  d’etudes  sur  les  equipes,  par  opposition  a  ceux  utilises  pour  la 
formation. 

L’examen  des  scenarios  a  ete  realise  en  fonction  d’un  certain  nombre  de  criteres.  D’apres  le  nombre 
de  criteres  auquel  il  repondait,  un  scenario  etait  juge  tres  pertinent,  pertinent  ou  plus  ou  moins 
pertinent.  Au  total,  37  scenarios  ont  ete  evalues,  et  15  ont  ete  juges  tres  pertinents.  L’evaluation  a 
permis  de  bien  cerner  les  criteres  qui  etaient  traites  de  faqon  coherente  par  les  etudes  sur  les  equipes 
et  ceux  qui  avaient  ete  negliges.  Il  a  done  ete  possible  d’etablir  un  plan  pour  creer  de  nouveaux 
scenarios  aux  fins  de  la  recherche  sur  les  equipes  a  DRDC.  Ce  plan  a  ete  examine  par  I’autorite 
scientifique  (AS),  et  une  orientation  detaillee  a  ete  fournie. 

Suivant  1’ orientation  donnee  par  FAS,  trois  scenarios  de  recherche  sur  les  equipes  dans  les  FC  ont 
ete  etablis  :  une  catastrophe  naturelle,  une  menace  terroriste  et  une  pandemic  de  grippe.  Ces 
scenarios  sont  tons  bases  sur  une  equipe  de  trois  personnes,  soit  trois  membres  de  COM  Canada, 
trois  membres  de  la  Force  operationnelle  interarmees  et  trois  personnes  venant  d’autres  ministeres 
(aux  niveaux  federal  et  provincial/local).  Les  trois  scenarios  permettent  d’examiner  tons  les  facteurs 
intervenant  dans  la  recherche  sur  les  equipes,  tel  que  defini  par  Sartori  et  ses  collaborateurs  (2006), 
et  ils  sont  tons  structures  pour  offrir  un  niveau  moyen  de  fidelite  et  un  niveau  eleve  de  controle.  Un 
modele  est  egalement  au  point  pour  detainer  les  scenarios  et  les  mesures  de  rendement  connexes. 
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Parallelement  aux  travaux  d’ elaboration  de  scenario,  un  modele  informatique  a  etc  evalue  en 
fonction  d’une  liste  de  criteres.  En  tout,  26  applications  de  modelisation  ont  etc  evaluees  suivant 
15  criteres.  Compte  tenu  des  besoins  du  PRA  sur  les  equipes,  il  a  etc  conclu  que  I’Environnement 
integre  de  modelisation  des  performances  (IPME)  etait  la  plate-forme  de  modelisation  informatique 
la  plus  adaptee.  Cette  conclusion  sous-entent  Eexistence  d’un  noyau  d’utilisateurs  experimentes  de 
riPME,  ce  qui  elimine  la  necessite  de  prevoir  une  «  nouvelle  »  courbe  de  I’apprentissage  pour  une 
«  nouvelle  »  application  de  modelisation  informatique.  Aucune  application  de  modelisation 
informatique  n’offrait  de  capacites  superieures  a  celles  de  I’lPME. 

Tant  r elaboration  de  scenarios  que  I’examen  des  modeles  informatiques  peuvent  deboucher  sur  de 
veritables  avantages  pour  les  EC.  Dans  le  premier  cas,  le  modele  informatique  peut  rapidement  et 
efficacement  aider  a  comprendre  1’ incidence  probable  des  interventions  basees  sur  1’ integration  des 
systemes  humains  (ISH)  et  a  cerner  les  domaines  auxquels  RDDC  devrait  s’interesser  en  priorite, 
qu’il  s’agisse  d’ISH  ou  de  recherche  sur  les  equipes.  Cela  pourrait  donner  lieu  a  d’importantes 
economies  de  temps  et  d’argent  et  faciliter  la  reussite  des  projets.  Apres  le  lancement  d’un 
programme  de  recherche  sur  les  equipes,  les  notions  ainsi  acquises  aideront  les  EC  a  structurer  les 
equipes,  a  leur  fournir  des  outils  qui  ameliorent  leur  rendement  et  a  comprendre  comment  corriger 
leurs  dysfonctionnements  (en  particulier  en  situation  de  stress). 

Ce  travail  a  etc  realise  aux  termes  du  contrat  W771 1-04791 1//001/TOR,  numero  de 
commande  7911-05.  L’AS  pour  ce  travail  est  M™  Renee  Chow. 
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Introduction 


Defence  Research  and  Development  Canada  (DRDC)  Toronto  is  in  the  process  of  developing  a 
team  research  platform  aimed  at  supporting  the  Canadian  Forces  (CF)  future  integrated  (rather 
than  air,  maritime,  or  land-only)  operations,  and  interoperability  with  allies.  Other  Government 
Departments  (OGDs)  and  Non-Government  Organizations  (NGOs).  To  support  the  development 
of  a  platform  for  running  team  experiments,  DRDC  Toronto  has  sponsored  two  studies  focusing 
on  the  existing  literature  on  teams,  and  team  research  platforms  used  around  the  world  and  the 
manner  in  which  they  are  implemented.  The  review  will  support  the  Crown  in  choosing  a 
specific  type  of  team  in  a  specific  work  context  as  the  focus  of  team  experiments  and  team 
modelling  to  be  conducted  in  a  multi-year  Applied  Research  Project  (ARP).  DRDC  Toronto  can 
apply  this  understanding  to  the  development  of  a  team  research  platform  that  adds  to  the  existing 
corpus  of  knowledge  about  teams,  and  builds  upon  the  best  aspects  of  the  extant  platforms  while 
avoiding  known  deficiencies  with  these  systems.  The  second  of  these  two  studies  also  reviews 
the  current  team  research  platforms  with  respect  to  the  scenarios  they  present  to  teams,  as  well  as 
reviewing  the  available  computational  models  that  have  been  used  to  model  and  predict  team 
performance.  The  direction  of  this  work  corresponds  to  the  DRDC  Science  and  Technology 
(S&T)  challenge  areas  PS-3:  Strategies  for  promoting  collaborative  behaviour  among  teams, 
agencies,  organisations  and  societies;  and  HU-2:  Human  systems  integration. 

In  pursuit  of  this  information,  DRDC  Toronto  has  sponsored  four  related  streams  of  work: 

1.  Conduct  a  literature  review  on  teams; 

2.  Conduct  a  literature  review  into  existing  platforms  for  running  team  experiments; 

3.  Review  team  research  scenarios  and  develop  domestic  scenarios  involving  the  CF, 
and, 

4.  Review  projects  from  around  the  world  describing  computational  models  of  teams. 

The  current  contract  addresses  the  latter  three  work  items  and  this  report  in  particular  describes 
the  last  two  work  items.  This  work  has  been  contracted  to  \lwmwsy stems  Incorporated®  as 
contract  no.  W7711-047911,  call  up  no. 7911-05.  The  Scientific  Authority  (SA)  for  this  work  is 
Renee  Chow. 

1.1  Objectives 

The  stated  objectives  of  the  information  in  this  report  are  twofold: 

To  support  the  identification  of  an  appropriate  context  for  conducting  new  Human-Systems 
Integration  (HSI)  research  on  teams  and  to  define  the  approaches  for  subsequent  experimental 
and  modelling  work  by: 

1 .  developing  scenario(s)  for  team  experiments  and  modelling  that  are  representative  of  the 
targeted  context; 
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2.  assessing  the  suitability  of  a  proposed  software  tool  for  modelling  teams  in  the  targeted 
context,  and  recommending  alternatives  as  appropriate. 

This  particular  phase  of  work  focused  on  an  in-depth  review  of  team  research  literature  for  the 

purposes  of  meeting  the  objectives.  In  order  to  achieve  this,  the  following  tasks  were  performed: 

1.1.1  Scenario  Development 

1 .  Meet  with  the  Scientific  Authority  to  select  a  niche  for  subsequent  research. 

2.  Review  relevant  documentation  to  arrive  at  a  thorough  understanding  of  the  teamwork  of 
interest. 

3.  Develop  scenarios  that  are  representative  of  the  teamwork  of  interest.  The  scenario  must  be 
able  to  support  a  team  experiment  involving  one  or  more  HSI  intervention(s).  It  must  also  be 
able  to  serve  as  a  basis  for  developing  a  computational  model  of  team  performance. 

4.  Propose  Measures  of  Performance  (MOPs)  and  Measures  of  Effectiveness  (MOEs) 
appropriate  for  an  experiment  using  the  scenarios. 

5.  Propose  one  or  more  augmentation(s)  or  variation(s)  to  the  scenario  developed  that  can  be 
used  for  training  subjects  for  the  team  experiment(s)  or  for  testing  the  generalisability  of  the 
team  model. 

1.1.2  Computational  Models 

1.  Review  IPME  manuals. 

2.  Review  open  literature  and/or  technical  reports  that  document  the  development  and/or 
validation  of  computational  models  of  team  performance  implemented  in  IPME. 

3.  Propose  how  the  scenario  developed  above  may  be  implemented  as  an  IPME  task  network 
model. 

4.  Propose  how  the  scenario  developed  above  may  be  implemented  in  an  IPME  crew  model. 

5.  Propose  how  the  scenario  developed  above  may  be  implemented  in  an  IPME  environment. 

6.  Identify  key  similarities  and  differences  between  the  proposed  implementation  and  previous 
implementations  of  team  models  in  IPME. 

7.  Propose  how  the  MOPs  and  MOEs  proposed  above  may  be  implemented  in  IPME. 

8.  Propose  the  use  of  an  existing  or  a  new  model  of  workload  in  IPME  for  modelling  the 
scenario  developed  above,  and  provide  the  rationale  for  choosing  this  workload  model. 

9.  Propose  changes  or  additions  to  IPME  that  will  increase  IPME’s  utility  for  modelling  the 
scenario  developed  above. 

10.  Propose  changes  or  additions  to  IPME  that  will  increase  IPME’s  usability  for  modelling  the 
scenario  developed  above. 
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11.  If  more  suitable  computational  modelling  platforms  are  identified,  recommend  one  or  more 
alternative(s)  to  IPME  that  will  be  more  appropriate  for  modelling  the  scenario  developed 
above  and  provide  the  rationale  for  the  recommendation(s). 

An  exhaustive  bibliographic  list  and  associated  literature  review  was  produced  under  a  separate 
contract  (Sartori  et  al,  2006),  but  was  used  extensively  to  shape  this  report.  In  particular, 
scenarios  and  computational  modelling  applications  uncovered  by  that  report  were  drawn  upon 
extensively  in  developing  this  work.  The  bibliographic  listing  will  not  be  replicated  in  full  in  this 
report.  Instead,  this  report  focuses  on  the  detailed  findings  from  the  literature  reviews 
undertaken  subsequent  to  the  initial  literature  review.  This  reflects  the  fact  that  the  initial 
literature  review  identified  the  most  prominent  literature  and  subsequent  searches  focused  on 
finding  additional  detail  and  contacts. 

The  method  adopted  for  each  aspect  of  this  work  (i.e.  scenario  development  and  computational 
models)  is  described  in  greater  detail  in  the  next  section. 
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Method 


The  methods  by  which  the  two  discrete  parts  to  this  work  were  addressed  are  described  at  a  high 
level  in  the  following  two  sections. 

2.1  General  approach  to  scenario  development 

In  developing  a  scenario  for  team  modelling,  three  main  steps  were  taken.  The  first  step  was  to 
review  the  outcome  of  preceding  phases  of  this  ARP.  The  second  step  was  to  conduct  a  general 
search  for  scenarios  in  support  of  team  experimentation.  The  last  step  was  to  establish  the 
desired  scope  and  breadth  of  the  scenario  to  be  generated.  This  was  done  through  a  number  of 
informal  deliverables  (described  below)  highlighting  the  direction  of  potential  scenarios  and  the 
subsequent  receipt  of  feedback  and  additional  guidance  from  the  SA. 

2.1.1  Literature  review 

The  literature  review  of  team  performance,  completed  in  the  first  phase  of  this  project  (Sartori  et 
al.,  2006),  identified  several  key  concepts  and  factors  that  relate  to  team  performance.  Early  in 
the  scenario  generation  process,  a  brainstorming  session  was  held  with  HSI®  team  members  who 
were  involved  in  the  literature  survey  (Sartori  et  al.,  2006),  the  experimental  platform  survey 
(Go  et  al.,  2006)  and  the  scenario  development  (current  task)  to  identify  critical  themes  relating 
to  team  performance  that  could  serve  as  a  basis  for  scenario  development.  As  a  result  of  this 
process,  four  main  themes  emerged  -  team  factors,  task  factors,  team  processes  and  team 
measures.  Team  factors  include  identifying  who  is  involved  in  the  team,  where  they  are  located 
and  what  the  inter-team  relationships  are.  Task  factors  describe  characteristics  inherent  to  the 
task,  such  as  task  complexity  and  workload  that  affect  team  performance.  Team  processes  refer 
to  those  aspects  (such  as  shared  knowledge)  that  emerge  out  of  group  interactions.  Lastly,  team 
measures  define  ways  of  evaluating  effects  on  team  performance.  These  four  main  themes  have 
been  decomposed  into  a  series  of  criteria.  Table  1  presents  a  list  of  the  criteria  that  were 
identified  as  important  to  defining  a  team  scenario.  Abbreviated  definitions  are  found  in  Table  1, 
detailed  explanations  of  these  terms  can  be  found  in  Sartori  et  al.  (2006). 


Table  1 :  List  of  Criteria 


1.TEAM  FACTORS 

1.1  Team  Size 

Small 

Less  than  or  equal  to  5  members. 

Medium 

6  to  19  members. 

Large 

20+  members. 

Teams-of-teams 

A  team  composed  of  two  or  more  teams  (sub-teams). 

1.2  Team  History 
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Ad  Hoc 

A  team  without  prior  history  that  is  assembled  in  response  to  a  particular  situation 
or  problem,  likely  to  be  from  diverse  backgrounds. 

Fixed 

Teams  that  have  worked  together  for  a  long  time  and  have  a  prolonged  history. 
Personnel  routinely  working  together. 

1.3  Physical  Distribution 

Distributed 

Geographically  distributed  teams  that  consist  of  individuals  in  different  locations, 
typically  understood  to  use  technologically  mediated  communications. 

Co-located 

Teams  that  are  located  in  the  same  physical  space. 

2.  TASK  FACTORS 

2.1  Task  Complexity 

Uncertainty 

Degree  of  predictability  or  confidence  associated  with  a  given  task. 

2.2  Workload 

Physical 

i.e.  fatigue 

Cognitive 

i.e.  decision  complexity 

Time  Pressure 

Urgency,  time  constraints 

2.3  Task  Interdependence 

Additive 

Individual  resources  are  summed  or  averaged  in  order  to  perform  the  task  (e.g., 
brainstorming  task). 

Conjunctive 

Performance  is  based  on  the  team's  lowest  performer  (e.g.,  assembly  line  task). 

Disjunctive 

Based  on  the  team's  highest  performer  (e.g.,  problem  solving  task). 

Discretionary 

Performed  by  self-managed  work  groups  as  they  have  the  authority  to 
autonomously  decide  how  to  divide  their  resources  (e.g.,  management  team 
initiating  organizational  initiatives). 

Pooled  interdependence 

Requires  less  coordination  as  sub-tasks  are  performed  separately  and  in  no 
specified  order. 

Sequential  interdependence 

Requires  linear  coordination,  such  that  subtasks  are  completed  in  a  specified 
sequence  (with  no  return  to  earlier  steps). 

Reciprocal  interdependence 

The  completed  subtask  of  one  team  member  becomes  the  input  for  the  second, 
and  the  second's  completed  subtask  becomes  the  input  for  the  third  and  so  on. 

3.  TEAM  PROCESSES 

3.1  Shared  Knowledge 

Team  Mental  Models 

Knowledge  about  roles/responsibilities,  abilities  of  one's  team  member(s). 

Task  Mental  Models 

Knowledge  about  task  requirements. 

3.2  Communication 

Implicit 

Voluntary  or  spontaneous  delivery  or  provision  of  information  without  an  explicit 
request  for  it.  (i.e.  push) 
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Explicit 

Offering  information  in  response  to  a  specific  request,  (i.e.  pull) 

Centralized  Network 

Messages  are  routed  through  a  key  member. 

Decentralized  Network 

All  group  members  have  a  potentially  equal  impact  on  communication  flow. 

Hierarchical  Network 

Similar  to  centralized  networks  in  that  there  is  a  key  member,  the  leader.  For 
example,  a  tier  immediately  beneath  the  leader  consists  of  more  junior  leaders, 
who  communicate  the  top  leader's  messages  to  the  bottom  tier. 

3.3  Team  Coordination 

Implicit 

Describes  the  ability  of  team  members  to  act  in  concert  without  the  need  for  overt 
communication. 

Explicit 

Requires  that  team  members  communicate  to  articulate  their  plans  and 
responsibilities,  i.e.  planning  of  roles,  responsibilities,  tasks,  and  procedures,  and 
communicating 

3.4  Team  Adaptability 

Monitoring 

Team  members  observe  and  assess  their  own  and  each  other's  performance  for 
the  purpose  of  remediating  deficient  task  work  and  teamwork  behaviours. 

Feedback 

Providing  information  to  other  team-mates  in  order  to  improve  or  correct  their 
behaviour. 

Backup  Behaviours 

Promoting  team  effectiveness  by  responding  in  a  timely  manner  to  other  team¬ 
mates  needs. 

3.5  Planning 

Resource  Allocation 

Division  of  team  resources  including  personnel,  time,  materials,  energy,  etc. 

4.  TEAM  MEASURES 

4.1  Outcome 

Automation 

i.e.  Computer 

Self-Report 

i.e.  Questionnaire 

Observer 

i.e.  SMEs 

4.2  Level  of  Analysis 

Individual  Performance 

Performance  considered  at  the  individual  level  of  analysis. 

Team  Performance 

Performance  considered  at  the  team  level. 

2.1.2  Scenario  search 

A  detailed  search  was  conducted  using  the  internet  and  library  system  at  the  University  of 
Toronto  to  identify  scenarios  used  to  date  in  team  research.  The  goal  in  conducting  this  search 
was  to  identify  example  scenarios  that  have  actually  been  used  for  team  research,  in  the  hopes  of 
better  understanding  the  progress  of  research  within  this  field.  The  scope  was  therefore  not 
limited  to  the  military  domain.  The  search  uncovered  a  large  number  of  team  related  scenarios, 
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however  many  were  not  representative  of  team  experiments.  The  majority  of  uneovered 
scenarios  were  those  that  facilitated  team  training.  It  was  therefore  decided  that  the  search  for 
team  scenarios  should  exclude  those  scenarios  used  for  the  purpose  of  team  training,  in  order  to 
reduce  the  number  of  scenarios  and  allow  for  a  greater  focus  on  team  experiments  involving  data 
collection  and  analysis.  Only  those  training  scenarios  that  were  associated  with  the  platform 
review  (Go  et  al.,  2006)  or  the  military  domain  are  included  in  this  report.  These  modifiers 
refined  the  search  and  resulted  in  a  manageable  number  of  scenarios.  A  total  of  37  scenarios 
were  identified. 

2.1.3  Mapping  of  criteria  to  scenarios 

The  scenario  search  led  to  the  identification  of  existing  experimental  scenarios  that  have  been 
used  in  the  area  of  team  research,  while  the  literature  survey  emphasized  criteria  that  could  be 
important  to  scenario  generation.  The  37  reviewed  scenarios  were  then  mapped  to  the  criteria 
listed  in  Table  1  to  uncover  emerging  patterns.  These  mappings  can  be  found  in  Annex  A.  The 
end  result  of  this  process  was  the  identification  of  the  ‘best’  or  most  relevant  aspects  of  scenarios 
used  for  team  research  in  the  past,  while  highlighting  those  factors  or  dimensions  that  have  been 
left  unexplored  by  current  research.  The  HSI®  team  used  this  approach  to  ensure  that  standard 
‘features’  of  team  research  are  maintained  while  emphasizing  the  opportunities  for  original  and 
groundbreaking  research. 

2.1.4  Creation  of  scenarios 

A  preliminary  plan  for  the  creation  of  scenarios  was  presented  to  the  SA  outlining  expected 
scenario  requirements,  two  potential  scenarios,  and  the  next  steps  for  this  work.  In  this  report,  it 
was  emphasized  that  the  scenarios  should  be  leveraged  from  the  results  of  the  platform  and 
literature  surveys,  as  well  as  general  knowledge  of  future  CF  requirements.  From  the  outset,  the 
SA  also  identified  that  the  scenario  should  include  joint,  interagency  and  interdisciplinary  teams 
performing  operational  level  activities.  Satisfying  the  SA’s  requirements  led  to  the  identification 
of  other  factors  that  should  be  incorporated  into  the  scenario.  For  example,  a  team  composed  of 
joint  CF  units,  multiple  agencies,  and  personnel  performing  distinct  roles  can  be  satisfied  by 
selecting  a  scenario  that  allows  for  teams-of-teams.  The  same  can  be  said  for  team  history,  team 
distribution,  and  team  size.  A  team  composed  of  sub  teams,  assembled  to  accomplish  a  specific 
goal  is  likely  to  be  an  ad  hoc  team.  Further,  a  scenario  that  fulfills  the  requirements  of 
interagency  and  interdisciplinary  will  likely  involve  small  or  medium  teams  working  from 
different  locations  (i.e.  distributed).  Details  of  the  scenario  plan  presented  to  the  SA  can  be 
found  in  Annex  B.  The  scenario  plan  also  described  potential  controlled  and  manipulated 
variables,  and  categories  of  MOPs.  Suggested  MOPs  came  from  the  literature  survey  and 
include  shared  knowledge,  communication  and  team  performance  measures.  To  demonstrate  the 
application  of  the  different  criteria  to  scenarios,  two  scenarios  were  described  by  answering 
questions  outlined  in  the  SOW.  The  first  scenario  was  based  on  a  real-life  domestic  joint 
operation  force  -  Winnipeg  Floods,  and  the  second  scenario  described  an  international  peace 
support  operation  involving  non-combatant  evacuation  using  a  multinational  joint  force 
headquarters.  The  last  section  of  the  scenario  plan  identified  steps  that  could  be  taken  once  a 
scenario  was  finalised. 
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After  reviewing  this  plan,  the  SA  proposed  a  revised  plan  that  emphasized  certain  aspects  and 
deemphasized  other  aspects  of  the  original  plan  with  regards  to  scenario  development.  The  main 
modification  was  to  consider  multiple  domestic  scenarios  and  not  to  develop  an  overseas 
scenario.  The  SA  specified  that  the  domestic  scenarios  should  model  a  natural  disaster,  a 
terrorist  threat  and  a  pandemic  scenario.  These  scenarios  would  be  broadly  sketched  out  rather 
than  defined  in  detail.  It  was  noted  that  the  Winnipeg  Floods  scenario  presented  in  the  original 
scenario  plan  could  be  used  as  the  natural  disaster  scenario.  Further,  the  SA  identified  key  teams 
(3  teams  of  2-3  members)  that  should  be  involved  in  each  of  the  domestic  scenarios.  Ideally  in 
each  scenario,  the  first  team  should  represent  a  high  level  of  command  within  DND,  the  second 
team  should  be  representative  of  a  lower  level  of  command  within  DND,  and  the  third  team 
should  include  players  from  an  OGD  (e.g..  Public  Safety  and  Emergency  Preparedness  Canada 
(PSEPC)).  The  SA  provided  a  modified  set  of  questions  to  be  answered  when  developing  the 
three  scenarios.  This  list  of  questions,  along  with  additional  modifications  that  were  requested  by 
the  SA  can  be  found  in  Annex  C. 

2.2  General  approach  to  evaluation  of  computational  modelling 
tools/approaches 

Three  converging  approaches  to  this  work  were  followed:  literature  review,  email  questionnaire 
and  domain  expert  interview.  The  information  gathered  was  used  to  assess  the  computational 
models  on  a  number  of  different  criteria  (described  below). 

2.2.1  Literature  Review 

Extensive  literature  search  were  conducted  through  the  Internet  (e.g.  via  Google)  and  the 
University  of  Toronto  library  system  (key  words  used  were  “computational  modeling”, 

“cognitive  modeling”,  “team  modeling”,  “team  performance”,  “team  process”).  A  total  of  26 
computational  models  were  identified  (see  Section  4).  The  reader  should  note  that 
“computational  model”  is  taken  to  mean  both  unique  applications  built  specifically  for  some 
specific  purpose  and  modeling  tools/environments  that  can  be  used  to  model  any  cognitive 
system.  During  this  research,  several  good  resources  for  reviews  and  comparisons  of 
computational  models  were  found  and  listed  in  the  references  to  this  report. 

2.2.2  Email  Questionnaire 

A  questionnaire  comprising  14  questions  was  sent  to  the  contact  person  (if  identified)  of  each 
model’s  developer  organization  (e.g.  research  group,  institute  or  company)  via  email.  In  total,  23 
questionnaires  were  sent,  excluding  only  3  models:  RESA,  PUMA  and  Wildfires  Fight 
Simulation  for  Training  (the  contact  persons  could  not  be  identified  for  these  models).  The 
questionnaire  sent  to  developer  organizations  is  presented  in  Annex  D,  with  the  responses  from 
the  developer  of  Brahms.  Questionnaires  sent  to  the  developer  organizations  of  the  other  22 
models  follow  the  same  format  except  that  the  name  of  the  model  being  surveyed  was  different. 
20  responses  were  received  form  the  following  15  models  (Affiliations  of  the  questionnaire 
responders  are  included  in  parenthesis): 
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IPME  (MA&D) 

IMPRINT  (MA&D) 

MIDAS  (NASA  Ames  Research  Center)  (two  responses) 

DDD  (Aptima,  Inc.) 

TOD  (Aptima,  Inc.) 

Soar  (use  Institute  for  Creative  Technologies,  Pennsylvania  State  University, 
Soar  technologies') 

Apex  (NASA  Ames  Research  Center)  (two  responses) 

D-OMAR  (BBN  Technologies) 

GLEAN  (University  of  Michigan,  Soar  Technologies^) 

Brahms  (NASA  Ames  Research  Center) 

JIMM:  (Naval  Air  Systems  Command) 

C3TRACE:  (U.  S.  Army  Research  Laboratory) 

CAST:  (Texas  A&M  University) 

EADSIM  (US  Army) 

Cogitoid  (Academy  of  Sciences  of  the  Czech  Republic) 

Although  responses  from  all  the  developer  organizations  were  not  received,  this  method  was  an 
efficient  method  of  gathering  information  quickly  in  the  sense  that  data  received  directly  from  the 
developers  and  researchers  of  each  model  are  more  reliable  and  convenient.  It  is  believed  that  if 
more  time  were  available  to  improve  the  approach  (e.g.  improving  the  way  of  contacting  the 
developer  organization,  including  a  description  of  each  criterion  in  the  questionnaire  etc.),  more 
(valid)  responses  would  be  received.  Regardless,  through  this  process,  a  contact  with  the 
developer  organization  of  a  model  has  been  made  for  in-depth  discussions  should  they  be 
required  at  a  later  date. 

2.2.3  Domain  Expert  Interview 

One  computational  modeling  expert  from  NASA  Ames  Research  Center  and  one  human 
performance  modeling  researcher  from  DRDC  Toronto  were  informally  interviewed  via 
telephone  for  their  opinions  on  the  selected  models  for  assessment.  This  was  also  a  good  way  to 
acquire  hidden  information  and  find  answers  for  difficult  questions  (e.g.  publicly  unavailable 
information  or  new  progress/trends  or  in-depth  expert  perspective).  If  time  and  resources  were 
available,  additional  expert  interviews  could  have  been  conducted  to  evaluate  functionality  and 


'  Note  that  Soar  has  multiple  developers 

^  Soar  Technologies  is  not  the  developer  of  GLEAN.  But  a  person  from  Soar  Technologies 
provided  opinions  on  GLEAN. 
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suitability  of  identified  computational  models,  especially  on  those  models  for  which  sufficient 
open  information  were  not  available. 

As  noted  above,  the  information  was  used  to  describe  the  computational  model’s  adequacy  in 
terms  of  a  set  of  criteria.  The  criteria  were  developed  from  the  statement  of  work  and  an 
understanding  of  what  characteristics  would  be  useful  to  discriminate  between  different  modeling 
applications.  These  criteria  are  listed  in  Table  2  below: 


Table  2:  Criteria  used  to  describe  Computational  Models 


Name  of  computational  models 

Model  specified  team  tasks 

Developer  organization 

Model  specified  team  interactions 

Measure  workload 

Model  HSI  interventions  of  interest 

Compatible  with  IPME 

Analyze  team’s  strategies 

Scenario  flexibility 

Analyze  team’s  performance 

Domain  independent 

Available  in  the  public  domain 

Model  team  as  entity 

Stable 

Model  team  as  a  group  of  individuals 

Real-time  computer  generated  forces 

Using  this  multi-dimensional  evaluation  of  the  available  computational  modeling  applications,  it 
would  be  possible  to  answer  the  questions  raised  in  the  original  statement  of  work. 
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Scenario  Development  Results 


The  results  stemming  from  scenario  development  are  subdivided  into  two  main  parts:  discussion 
of  the  results  of  the  scenario  search  and  the  presentation  of  scenarios  generated  specifically  for 
this  ARP. 

3.1  Scenario  Review  Results 

The  scenario  search  led  to  the  identification  of  37  scenarios  relevant  to  team  experiments. 
Fourteen  of  the  reviewed  scenarios  are  associated  with  platforms  described  in  the  previous  report 
(Go  et  al.,  2006).  Platforms  and  scenarios  can  be  tied  together  in  a  number  of  ways.  A  platform 
can  be  used  to  run  multiple  scenarios  or  a  single  scenario  can  be  used  by  multiple  platforms. 
Therefore  the  relation  between  scenarios  and  platforms  is  not  a  one  to  one  mapping. 

Each  scenario  included  in  this  report  was  carefully  reviewed  and  evaluated.  Of  the  37  scenarios, 
15  were  deemed  ‘highly  relevant’,  9  were  judged  ‘relevant’  and  13  were  deemed  ‘somewhat 
relevant’  as  depicted  in  Table  3  below.  Relevance  rankings  are  based  on  a  9  point  scale  that  is 
subdivided  into  three  categories:  originality,  expandability  and  complexity.  Originality  refers  to 
the  how  original  the  scenario  is,  expandability  refers  to  a  scenario’s  ability  to  incorporate  more 
variables,  and  complexity  refers  to  the  number  of  variables  involved  in  the  scenario.  Each 
scenario  can  score  a  maximum  of  three  points  in  any  of  the  categories  and  a  minimum  of  one 
point  in  each  of  the  categories.  A  scenario  was  ranked  ‘highly  relevant’  when  it  scored  the 
maximum  number  of  points  in  at  least  two  of  the  categories.  A  scenario  was  ranked  ‘relevant’ 
when  it  scored  a  perfect  score  in  one  of  the  three  categories.  Lastly,  a  scenario  was  ranked 
‘somewhat  relevant’  when  it  did  not  score  a  perfect  three  in  any  of  the  categories.  It  is  important 
to  remember  that  all  scenarios  described  in  this  report  have  been  included  because  they  are 
suitable  for  team  research. 

Twenty-five  of  the  reviewed  scenarios  were  at  the  operational  level  and  the  majority  of  scenarios 
included  in  this  report  were  used  for  research  purposes  (N  =  31).  The  categories  ‘level  of 
activity’  and  ‘purpose’  were  not  treated  as  mutually  exclusive  categories  since  a  scenario 
sometimes  involved  strategic  and  operational  level  components,  and  similarly  is  used  for  both 
training  and  research  purposes.  Therefore  the  summed  totals  for  these  categories  exceed  the 
number  of  reviewed  scenarios. 
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Table  3:  Overview  of  Reviewed  Scenarios 


Evaluation  Criteria 

Number  of  Scenarios  that  met  Evaluation  Criteria 

Relevance 

Highly  relevant 

15 

Relevant 

10 

Somewhat  relevant 

12 

Total 

37 

Level  of  activity 

Strategic 

4 

Operational 

25 

Tactical 

12 

Total 

41 

Purpose 

Training 

10 

Research 

31 

Total 

41 

The  next  step  in  evaluating  the  team  experiment  seenarios  was  to  doeument  which  criteria  the 
scenarios  tended  to  cover  and  which  criteria  have  been  unexplored  by  previous  research.  The 
mappings  of  each  scenario  to  the  criteria  can  be  found  in  Annex  A.  It  is  important  to  note  that 
when  scenarios  were  reviewed,  if  a  criterion  was  explicitly  described  or  alluded  to  (i.e.  the 
reader  could  infer  that  a  factor  was  addressed,  then  that  scenario  is  assumed  capable  of 
supporting  that  criterion.  In  the  latter  case,  the  mapping  of  scenarios  to  the  criteria  required 
some  judgment  on  the  part  of  the  evaluator.  The  specific  words  chosen  to  describe  a  criterion 
are  not  necessarily  the  same  terms  that  are  used  in  the  scenario  descriptions.  Therefore  the 
evaluator  had  to  interpret  and  judge  meaning  before  answering  ‘yes’  to  a  criterion,  especially 
when  the  exact  term  did  not  appear  in  the  scenario  description.  To  minimize  discrepancies  in 
subjective  judgement  and  maintain  consistency  in  mappings,  the  evaluator  was  given  clear 
definitions  of  each  criterion  beforehand  (Table  1),  and  all  37  scenarios  were  mapped  by  the  same 
evaluator.  Descriptions  of  each  scenario  can  be  found  in  Annex  E. 

It  is  valuable  to  understand  the  context  in  which  the  criteria  are  used.  A  criterion  can  be  either 
an  independent  or  dependent  variable.  An  independent  variable  is  that  which  is  controlled  for  by 
the  experimenter,  while  a  dependent  variable  is  that  which  may  vary  and  what  the  experimenter 
is  tracking  during  a  scenario.  When  the  reviewer  was  unable  to  determine  if  a  criterion  was  an 
independent  or  dependent  variable  then  the  box  ‘unknown’  was  populated,  indicating  that  the 
criteria  was  used  in  the  scenario  but  the  conditions  are  unclear. 

If  a  criterion  is  left  blank  during  the  mapping  process  then  this  could  mean  one  of  two  things, 
that  either  that  level  of  detail  is  not  given  (i.e.  the  reviewer  did  not  have  enough  information  to 
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make  a  conclusion)  or  the  scenario  is  incapable  of  supporting  that  feature.  Therefore, 
frequencies  tabulated  across  scenarios  should  only  be  considered  indicative. 

Table  4  below  illustrates  the  clusters  and  gaps  of  knowledge  for  team  factors. 


Table  4:  Team  Factors  in  Reviewed  Scenarios 


Evaluation  Criteria 

Number  of  Scenarios  that  met  Evaluation  Criteria 

Independent  Variable 

Dependent  Variable 

Unknown 

Total 

1.1  Team  Size 

Small 

27 

27 

Medium 

1 

1 

Large 

2 

2 

Teams-of-teams 

12 

12 

Total 

42 

1.2  Team  History 

Ad  Hoc 

10 

10 

Fixed 

26 

26 

Total 

363 

1.3  Physical  Distribution 

Distributed 

18 

18 

Co-located 

18 

18 

Total 

364 

Table  4  shows  that  in  each  of  the  reviewed  scenarios  team  factors  were  considered  independent 
variables.  This  means  that  team  factors  tended  to  be  selected  and  controlled  for  by 
experimenters.  The  findings  suggest  that  team  factors  have  generally  focused  on  small  (N=27), 
fixed  (N=26),  and  distributed  (N=  18)  and  co-located  (N=  18)  teams.  Conversely,  a  small 
number  of  scenarios  have  supported  large  (N=2)  teams. 

The  team  size  category  is  not  mutually  exclusive.  For  example  team  size  can  be  small,  medium, 
or  large  but  also  teams-of-teams.  If  a  scenario  presented  a  case  where  three  teams  of  four,  eight 
and  twelve  members  are  working  together,  then  the  checkboxes  small,  medium  and  teams-of- 
teams  are  populated. 


^  Scenario  31,  Air  Defense  Mission  of  an  AWACS  using  platform  C3STARS,  did  not  speedy  details 
regarding  team  history. 

Seenario  35,  Team  C2  Task  did  not  speeify  the  physieal  distribution  of  its  team. 
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When  the  scenario  involves  teams-of-teams,  team  history  is  referring  to  the  history  between  the 
teams.  When  the  scenario  involves  a  single  team,  then  team  history  is  referring  to  the  history 
between  members  of  the  team  (in  real-life).  Reviewers  applied  the  fixed  criterion  to  scenarios 
where  teams-of-teams  or  individual  participants  are  expected  to  regularly  work  together.  For 
example,  a  scenario  involving  teams  of  fire,  police  and  ambulance  services  are  considered  fixed 
teams.  It  is  likely  that  these  teams  work  together  regularly  and  therefore  have  in  place  set 
procedures.  The  criterion  ad  hoc  is  reserved  for  those  scenarios  that  involved  an  unusual 
combination  of  teams.  For  example  in  the  ATC  pilot  scenario  (#33),  military  air  crews  flew  two 
simulated  missions.  During  one  of  the  simulated  flights,  ‘something  went  wrong’  and  thus 
required  the  team  to  communicate  and  cooperate  with  persons  that  don’t  normally  work  together. 
Scenarios  are  considered  ad  hoc  when  the  teams  were  formed  in  response  to  unusual  emergency 
situations. 

The  physical  distribution  of  teams-of-teams  is  referring  to  the  location  of  different  teams,  while 
the  physical  distribution  of  a  single  team  is  referring  to  the  location  of  team  members.  Reviewed 
scenarios  used  both  distributed  (N=  18)  and  co-located  teams  (N  =  18).  A  scenario  was  judged  to 
include  distributed  teams  when  participants  or  teams  could  not  communicate  face  to  face  but  did 
so  via  technology. 

Table  5  below  illustrates  the  clusters  and  gaps  of  knowledge  for  task  factors. 


Table  5:  Task  Factors  in  Reviewed  Scenarios 


Evaluation  Criteria 

Number  of  Scenarios  that  met  Evaluation  Criteria 

Independent  Variable 

Dependent  Variable 

Unknown 

Total 

2.1  Task  Complexity 

Uncertainty 

19 

19 

Total 

19 

2.2  Workload 

Physical 

2 

2 

Cognitive 

8 

6 

1 

15 

Time  Pressure 

7 

2 

1 

10 

Total 

27 

2.3  Task  Independence 

Additive 

33 

1 

34 

Conjunctive 

2 

2 

Disjunctive 

1 

1 

Discretionary 

4 

4 

Pooled 

2 

2 

Sequential 

1 

1 

Reciprocal 

Total 

44 
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Findings  suggest  that  most  of  the  reviewed  scenarios  incorporated  uncertainty  (N=  19),  cognitive 
workload  (N=  15)  and  additive  tasks  (N=34).  Conversely,  a  small  number  of  scenarios  have 
explored  conjunctive,  disjunctive,  discretionary,  pooled,  sequential  and/or  reciprocal  tasks. 
Although  a  small  number  of  scenarios  have  addressed  physical  workload,  we  do  not  recommend 
that  this  ARP  address  physical  workload  since  it  is  less  significant  to  future  CF  requirements  at 
the  operational  level. 

Uncertainty  was  used  as  an  independent  variable  by  19  of  the  reviewed  scenarios.  This  is  not  to 
say  that  the  remaining  18  scenarios  do  not  incorporate  uncertainty  of  any  degree.  Although  it  is 
expected  that  all  scenarios  have  some  level  of  uncertainty,  it  was  usually  the  case  that  scenario 
descriptions  did  not  emphasize  uncertainty  as  a  variable  and  therefore  the  reviewer  was  unable  to 
confidently  draw  such  a  conclusion. 

Physical  workload  was  used  as  a  dependent  variable,  and  cognitive  workload  and  time  pressure 
were  used  as  both  independent  and  dependent  variables  in  the  reviewed  scenarios.  A  scenario 
could  impose  both  physical  and  cognitive  workload,  as  well  as  time  pressure  on  team  members. 
Therefore  the  category  workload  does  not  comprise  mutually  exclusive  options.  It  is  difficult  to 
gauge  whether  a  scenario  has  time  pressure,  so  a  scenario  would  answer  ‘yes’  to  this  criterion 
when  time  pressure  was  indicated  as  an  important  variable. 

Task  interdependence  describes  how  members  interact  and  depend  on  each  other  in  order  to 
attain  a  goal.  The  majority  of  scenarios  consisted  of  additive  tasks,  meaning  that  individual 
resources  are  summed  or  averaged.  The  category,  task  interdependence,  is  not  necessarily 
comprised  of  mutually  exclusive  options.  For  example,  a  task  can  be  both  conjunctive  and 
discretionary.  A  task  of  this  nature  would  involve  work  performed  by  self  managed  work  groups 
(discretionary)  with  the  team’s  lowest  performer  as  responsible  for  the  performance  of  the  team 
(conjunctive). 

Table  6  below  illustrates  the  clusters  and  gaps  of  knowledge  for  team  processes. 


Table  6:  Team  Processes  in  Reviewed  Scenarios 


Evaluation  Criteria 

Number  of  Scenarios  that  met  Evaluation  Criteria 

Independent  Variable 

Dependent  Variable 

Unknown 

Total 

3.1  Shared  Knowledge 

Team  mental  model 

3 

4 

5 

12 

Task  mental  model 

4 

4 

5 

14 

Total 

26 

3.2  Communication 

Implicit 

2 

10 

14 

26 

Explicit 

7 

15 

12 

Total 

38 
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Evaluation  Criteria 

Number  of  Scenarios  that  met  Evaluation  Criteria 

Independent  Variable 

Dependent  Variable 

Unknown 

Total 

Centralized  network 

6 

2 

8 

Decentralized 

network 

6 

4 

2 

12 

Hierarchical  network 

6 

1 

2 

9 

Total 

29 

3.3  Coordination 

Implicit 

4 

8 

11 

23 

Explicit 

4 

12 

16 

Total 

39 

3.4  Team  Adaptability 

Monitoring 

3 

3 

Feedback 

3 

7 

10 

Back-up  behaviour 

1 

1 

Total 

14 

3.5  Planning 

Resource  Allocation 

14 

14 

Total 

14 

With  regards  to  team  processes,  findings  suggest  that  the  reviewed  scenarios  have  focused  on 
task  mental  models  (N=  14),  team  mental  models  (N  =  12),  implicit  communication  (N=26),  a 
decentralized  network  (N  =  12),  implicit  coordination  (N  =  23),  feedback  (N  =  10),  and  resource 
allocation  (N  =  14).  Only  one  of  the  37  scenarios  addressed  the  criterion  back-up  behaviour. 

The  category  shared  knowledge  consists  of  team  mental  models  and  task  mental  models.  These 
two  types  of  mental  models  are  not  mutually  exclusive.  Both  criteria  are  used  as  independent 
(presented  as  part  of  the  scenario)  and  dependent  (measured  and  analyzed)  variables  in  the 
reviewed  scenarios.  No  scenario  used  one  mental  model  as  the  independent  variable  and  the 
other  mental  model  as  the  dependent  variable.  When  both  team  mental  models  and  task  mental 
models  were  integral  to  a  scenario,  they  were  classified  together  as  either  independent  variables, 
dependent  variables  or  unknown. 

Implicit  communication  is  the  voluntary  or  spontaneous  delivery  of  information  while  explicit 
communication  is  the  offering  of  information  in  response  to  a  specific  request.  These  two  types 
of  communication  are  not  mutually  exclusive.  A  scenario  answered  ‘yes’  to  this  criterion  when 
information  push  (implicit)  and  information  pull  (explicit)  are  allowed  or  exist  within  the 
scenario.  In  the  reviewed  scenarios,  implicit  and  explicit  communications  were  both  independent 
and  dependent  variables.  No  scenario  used  one  type  of  communication  as  the  independent 
variable  and  the  other  type  of  communication  as  the  dependent  variable. 


18 


Team  Modelling:  Scenarios  and  Computational  Models  ^iumamystems 


Twelve  of  the  reviewed  scenarios  had  teams  functioning  in  a  decentralized  network,  nine  in  a 
hierarchical  network  and  eight  in  a  centralized  network.  The  communication  networks  were 
classified  as  independent  and  dependent  variables.  Some  scenarios  (#19,  26,  32,  and  34)  studied 
the  effects  of  different  communication  networks  on  team  performance. 

Implicit  coordination  is  team  member’s  ability  to  act  in  concert  without  the  need  for  overt 
communication  and  explicit  coordination  requires  team  members  to  communicate  and  articulate 
their  plans  and  responsibilities.  In  reviewed  scenarios,  implicit  coordination  was  used  as  the 
independent  and  dependent  variable  while  explicit  coordination  was  used  only  as  the  dependent 
variable.  Implicit  and  explicit  coordination  are  not  mutually  exclusive. 

Team  adaptability  consists  of  monitoring,  feedback  and  back-up  behaviours.  Monitoring  is  when 
team  members  observe  and  assess  their  own  and  each  other’s  performance.  Feedback  is 
providing  information  to  other  team  members  in  order  to  improve  or  correct  behaviours,  and 
backup  behaviour  is  promoting  team  effectiveness  by  responding  in  a  timely  manner  to  team 
members’  needs.  Monitoring  and  back-up  behaviours  were  used  in  the  reviewed  scenarios  as 
dependent  variable.  Feedback  was  used  as  both  independent  and  dependent  variables.  These 
behaviours  are  not  mutually  exclusive. 

In  14  of  the  37  scenarios  resource  allocation  was  a  dependent  variable.  In  many  of  the  scenarios, 
resource  allocation  was  tracked  by  a  computer  and  analyzed  by  graphic  tools. 

Table  7  below  illustrates  the  manner  in  which  the  reviewed  experimental  scenarios  were 
measured. 


Table  7:  Measures  of  Performance  in  Reviewed  Scenarios 


Evaluation  Criteria 

Number  of  Scenarios  that  met  Evaluation  Criteria 

4.1  Outcome 

Automation 

28 

Self-Report 

12 

Observer 

13 

Total 

53 

4.2  Level  of  Analysis 

Individual 

11 

Team 

35 

Total 

46 

These  findings  suggest  that  most  experimental  scenario  outcomes  were  measured  automatically 
(N=28)  and  the  level  of  analysis  tended  to  be  teams  (N  =  35).  The  method  employed  to  measure 
the  dependent  variable  is  selected  by  experimenters.  Automation,  self-report  and  observer  are 
not  mutually  exclusive  criteria.  Similarly,  the  level  of  analysis  can  be  both  at  the  individual  or 
team  level. 
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3.2  Team  Research  Scenarios 


Having  received  detailed  guidance  from  the  SA  regarding  the  emphases  of  the  scenario  to  be 
developed,  three  scenarios  were  developed.  Of  these,  two  were  fictitious  and  incorporated  the 
most  desirable  features  identified  in  the  review  of  previous  team  research  scenarios.  The  first 
scenario  is  based  on  real  events  that  occurred  during  the  Winnipeg  Floods  of  1997.  This 
scenario  is  also  the  subject  of  large-scale  joint  exercises  at  the  operational  and  strategic  level  in 
and  around  the  National  Capital  Region. 

At  a  high  level,  the  SA  provided  significant  direction  regarding  the  structure  of  the  teams  to  be 
represented  in  the  scenario  and  the  subjects  of  the  scenarios.  Further  to  that,  the  SA  provided 
guidance  regarding  the  factors  she  would  like  described  within  each  scenario. 

3.2.1  Team  Structure  -  General  Approach 

For  expediency  in  an  experimental  environment  and  due  to  a  lack  of  publicly  available 
information,  the  team  structures  adopted  for  the  scenarios  represent  an  abstraction  of  how  teams 
from  different  organizations  might  actually  operate  if  an  emergency  situation  were  to  occur  in 
Canada.  Accordingly,  any  reference  to  specific  team  members  and  the  flow  of  information 
between  teams  is  an  approximation  of  how  the  Department  of  National  Defence  (DND)  and 
Other  Government  Departments  (OGDs)  could  operate. 

The  scenarios  were  developed  with  the  goal  of  emphasizing  interactions  within  and  between 
teams  at  the  operational  level.  Therefore  three  teams  were  chosen.  One  team  is  representative 
of  a  high  level  of  DND  Command  such  as  CanadaCom,  the  second  team  is  a  lower  level  of  DND 
Command  such  as  a  Regional  Joint  Task  Force,  and  the  third  team  represents  an  OGD  such  as 
PSEPC.  Choosing  these  teams  to  participate  in  the  scenarios  fulfilled  the  requirements  of  an 
environment  that  supports  teams-of-teams  in  a  joint,  interagency  and  distributed  setting.  The 
expected  relationship  between  teams  is  depicted  in  Figure  1,  whereby  arrows  represent  possible 
lines  of  communication. 
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Figure  1:  Proposed  structure  of  teams-of-teams  for  team  research  scenario 


The  lines  of  communication  between  teams  are  unclear  because  of  a  lack  of  publicly  available 
information  since  the  CanadaCom  organization  the  regional  joint  task  forces  have  only  gained 
operational  readiness  in  February  of  2006.  Further,  there  is  a  lack  of  publicly  available  detailed 
examples  demonstrating  how  CanadaCom  would  execute  its  responsibilities  in  a  manner  that  is 
coordinated  with  other  local,  provincial  and  federal  government  bodies  during  a  domestic 
emergency  situation.  However  it  is  expected  that  when  teams  would  work  together  in  response 
to  an  emergency  domestic  situation  the  following  information  would  be  regularly  communicated 
and  updated: 

•  Who:  who  is  the  requesting  agency  and  what  are  the  points  of  contact 

•  What:  what  types  of  support  is  required  to  accomplish  the  mission 

•  When:  when  is  the  support  required  and  what  the  desired  duration  is 

•  Where:  what  is  the  specific  location  for  the  proposed  operation 

•  Why:  statement  as  to  why  support  is  needed.  This  will  also  help  in  determining  what 
types  of  resources  are  required. 
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Although  the  lines  of  eommunication  are  unclear  at  this  point,  it  is  expected  that  the  lines  of 
communication  between  CanadaCom,  the  Regional  Task  Force  and  the  OGD  would  be  preserved 
from  scenario  to  scenario  since  the  goals  of  the  CF  transformation  and  the  subsequent  inception 
of  CanadaCom  is  a  centralized  organizational  scheme.  The  updated  CF  structure  is  expected  to 
clearly  delineate  authority,  responsibility,  chain  of  command  and  accountability.  Further,  the 
new  CF  structure  emphasizes  a  clear  separation  of  strategic  and  operational  responsibilities.  The 
strategic  level  of  command  is  responsible  for  strategic  decision  making,  policy,  strategic 
planning,  resource  allocation,  processes  and  strategies.  The  operational  level  of  command  is 
responsible  for  the  execution  of  standards  set  out  by  the  strategic  level. 

The  Commander  of  CanadaCom  is  to  report  directly  to  the  Chief  of  Defence  Staff  (CDS). 
CanadaCom  is  further  divided  into  six  task  forces  based  on  geography.  Each  regional  joint  task 
force  is  responsible  for  all  domestic  and  contingency  operations  within  their  region.  The  six 
regional  joint  task  forces  are: 

•  Pacific  Command  (British  Columbia), 

•  Prairie  Command  (Alberta,  Saskatchewan,  Manitoba), 

•  Central  Command  (Ontario), 

•  Eastern  Command  (Quebec), 

•  Atlantic  Command  (New  Brunswick,  Nova  Scotia,  Prince  Edward  Island,  Newfoundland 
and  Labrador), 

•  and  Northern  Command  (Yukon  Territories,  Northwest  Territories  and  Nunavut). 

The  operational  level  of  command  (CanadaCom  and  Regional  Joint  Task  Forces)  is  responsible 
for  employing  forces  to  attain  strategic  objectives  in  a  theatre  or  area  of  operations  through  the 
design,  organization  and  conduct  of  campaigns  and  major  operations.  Further,  at  the  operational 
level,  activity  is  conceived  and  conducted  as  one  single  concentrated  effort,  rather  than  according 
to  the  various  environmental  forces. 

CanadaCom  and  Regional  Joint  Task  Forces  are  likely  to  communicate  via  a  CF  liaison.  It  is 
unclear  as  to  how  the  CF  would  communicate  with  OGDs  and  vice  versa,  but  it  is  also  expected 
that  this  would  be  facilitated  by  a  liaison  officer.  Within  each  of  the  three  teams,  team  members 
can  communicate  directly,  therefore  the  relationships  between  and  within  CanadaCom,  a 
Regional  Joint  Task  Force  and  an  OGD  allows  for  horizontal  and  vertical  communication. 

The  structure  described  above  does  not  present  detailed  descriptions  of  how  CanadaCom,  a 
regional  joint  task  force  and  the  OGD  would  operate.  This  is  due  to  a  lack  of  publicly  available 
information  as  well  as  to  preserve  flexibility  within  the  scenarios.  The  above  structure  results  in 
a  medium  level  of  fidelity  which  affords  some  degree  of  experimental  control.  Ideally,  the  goal 
is  to  balance  experimental  control  with  realism. 

3.2.2  Scenario  Events  -  General  Approach 

The  above  section  presents  a  model  of  how  teams  could  be  structured  within  a  scenario.  In 
addition  to  this,  it  is  valuable  to  conceive  of  a  possible  flow  of  events  that  could  take  place  during 
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a  scenario.  This  section  presents  a  model  of  how  a  seenario  and  its  sub  events  could  be  staged  by 
researchers. 

From  the  onset,  the  SA  specified  an  interest  in  multiple  domestic  scenarios  with  less  granularity, 
sketched  out  rather  than  fully  instantiated.  Therefore  this  section  suggests  a  systematic  method 
for  developing  detailed  scenarios  (although  the  detailed  scenarios  is  not  developed),  ensuring  that 
relevant  components  ean  be  cultivated  to  a  level  sufficient  to  run  an  experiment.  If  this  were  not 
done,  any  experiment  would  always  run  the  risk  of  failing  due  to  some  incongruity  in  the 
scenario. 

To  demonstrate  the  sorts  of  detail  that  would  need  to  be  specified  in  a  final  scenario,  eonceptual 
diagrams  are  presented  below:  one  is  a  high  level  view  (Figure  2)  and  one  is  a  lower  level  view 
(Figure  3). 


Modifying  and  distracter  inputs 


Modifying  and  distracter  inputs 


Figure  2:  Overview  of  scenario  flow 
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Figure  3:  General  task  structure  throughout  scenario 

The  overview  of  the  scenario  (Figure  2)  shows  that  each  scenario  must  have  a  trigger  point.  This 
could  be  a  request  from  a  regional  authority,  intelligence  information,  or  some  actual  event.  An 
option  could  be  to  direct  the  trigger  point  to  the  OGD  who  must  then  decide  if  and  when  to 
involve  DND.  The  trigger  point  will  need  to  be  adequately  defined  and  delivered  to  the  recipient 
as  some  sort  of  input  (to  account  for  situation  awareness  needs,  this  trigger  point  will  most  likely 
take  the  form  of  a  written  brief  with  accompanying  maps  and/or  animations).  The  teams 
participating  in  the  experiment  will  need  to  plan  the  actions  of  their  resources  to  resolve  the 
scenario.  Each  scenario  will  have  a  number  of  conditions  that  will  need  to  be  satisfied  before  the 
scenario  will  be  deemed  over.  Although  this  point  may  become  increasingly  apparent  to 
participants,  it  will  be  the  responsibility  of  the  lead  researcher  to  call  a  halt  to  the  experiment. 

The  other  feature  of  the  scenario  overview  is  the  incorporation  of  modifying  or  distracter  inputs 
throughout  the  scenario.  It  was  felt  that  these  additional  inputs  are  necessary  to  ensure  that 
experimental  results  mirror  real-world  situations.  These  inputs  do  not  compromise  experimental 
control.  Indeed,  they  are  under  the  complete  control  of  the  researchers  and  can  be  planned  in 
advance  and  deployed  as  appropriate.  These  inputs  are  likely  to  directly  serve  the  research 
purposes  of  the  team  running  the  experiment.  Because  these  inputs  will  either  modify  the  task 
performed  by  the  team  (e.g.  intelligence  changes  the  area  of  operations),  or  distract  the  team 
from  the  task  at  hand  (e.g.  media  request  for  a  briefing)  different  team  processes  and  factors  can 
be  provoked  and  exercised.  Thus,  when  developing  the  detailed  scenario,  the  start  and  end 
points  must  be  defined,  as  must  the  planned  inputs. 

Figure  3  shows  the  detail  of  how  each  input  (both  the  start  point  and  modifying  or  distracter 
inputs)  will  be  acted  upon  and  how  it  may  further  modify  the  scenario.  The  team  or  the  team 
member  will  receive  an  input  via  some  route  (e.g.  paper,  telephone,  etc.).  This  will  trigger  some 
activity  by  the  team  and  its  members,  resulting  in  an  output.  That  output  will  be  received  by 
someone,  either  internal  to  the  team  or  external  to  the  team  (e.g.  other  teams  in  the  experiment. 
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peripheral  organisations,  subordinate  formations,  etc.)  thus  triggering  further  activities. 

However,  this  may  also  result  in  feedback  to  the  team,  which  itself  is  another  input  which  may 
modify  the  task.  The  output  could  also  just  be  some  step  (internal  to  the  team)  on  the  way  to 
achieving  a  broader  goal  for  the  team.  When  developing  the  detailed  scenario,  the  specific  inputs 
must  be  defined  as  should  the  expected  outputs.  Further,  the  possible  expected  feedback  must  be 
defined  so  the  scenario  does  not  move  in  unanticipated  directions.  With  this  detailed 
understanding,  it  will  be  possible  to  develop  detailed  scenario-based  MOPs. 

Feedback  to  the  team  could  be  provided  by  simulation  players  or  an  overall  ‘Experiment 
Manager’.  This  approach  is  used  by  the  large-scale  simulations  undertaken  by  the  Army 
Simulation  Centre  (ASC).  The  ASC  employs  many  contractors  and  actual  subordinate  units  to 
operate  the  ATHENA  tactical  system,  implement  the  plan  at  a  tactical  level,  and  feed  back 
information  into  the  planning  staffs.  Given  that  scenario  development  will  identify  the  desired 
output  based  on  an  input  that  occurs  at  a  specific  time,  if  the  output  is  not  produced  in  a  timely 
manner,  feedback  could  be  negative  (e.g.  casualty  rate  rises,  flood  waters  pass  a  critical  dam, 
etc.). 

3.2.3  Scenarios 

The  SA  requested  that  three  scenarios  be  developed  at  a  coarse  level  of  detail:  a  natural  disaster; 
a  terrorist  threat;  and  a  pandemic.  The  SA  requested  that  these  scenarios  be  described  according 
to  a  number  of  dimensions.  Each  scenario  is  briefly  described  below  and  then  the  specific 
dimensions  of  each  scenario  are  described  in  Table  8. 

3. 2.3.1  Scenario  1  -  Natural  Disaster:  Winnipeg  Floods 

The  first  scenario  is  the  Winnipeg  Floods.  The  Red  River  Flood  of  1997  was  a  major  flood  that 
occurred  in  April  and  May  1997,  along  the  Red  River  in  North  Dakota,  Minnesota,  and 
Manitoba.  It  was  the  most  severe  flood  of  the  river  since  1826.  The  flood  reached  throughout 
the  Red  River  Valley,  affecting  the  city  of  Winnipeg.  Operation  Assistance  was  the  name  given 
by  the  Canadian  Forces  for  military  support  to  the  civil  authorities  during  the  flooding. 

Organizations  that  could  be  involved  (as  well  as  DND): 

•  City /Provincial  Departments  -  Ambulance,  Fire,  City  of  Winnipeg  Police  Service, 
Harbour  Patrol,  Social  Services,  Emergency  Preparedness  and  Coordination  Committee 
(EPCC),  Centre  for  Emergency  Preparedness  and  Response  (CEPR) 

•  Federal  -  Royal  Canadian  Mounted  Police  (RCMP),  Public  Safety  and  Emergency 
Preparedness  Canada  (PSEPC),  Emergency  Public  Information  Team  (EPIT),  Public 
Health,  Public  Utilities,  Public  Works 

•  Outside  Agencies  -  Media 

•  Private  Sector  -  United  Way,  Meals  on  Wheels,  Red  Cross,  Salvation  Army 
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3.2.3.2  Scenario  2  -  Terrorist  Threat 

This  scenario  is  fictitious  but  would  involve  an  intelligence  report  that  a  terrorist  organisation  is 
to  launch  an  imminent  attack  on  the  financial  district  of  Toronto.  The  report  is  not  specific 
enough  to  deploy  forces  to  counter  the  threat.  Rather,  the  warning  is  such  that  the  precise  nature 
of  the  threat  is  unknown,  and  the  various  teams  must  mobilise  resources  to  address  a  wide  range 
of  potential  eventualities. 

Organizations  that  could  be  involved  (as  well  as  DND): 

•  City/Provincial  Departments  -  Ambulance,  Fire  (Hazmat),  Ontario  Provincial  Police 
(OPP),  Toronto  Police  Service  (TPS),  Toronto  Police  Bomb  Squad 

•  Federal  -  RCMP,  PSEPC,  Canadian  Security  Intelligence  Service  (CSIS),  National 
Security  Investigation  Seetions  (NSIS),  Integrated  National  Security  Enforcement  Teams 
(INSETs)  (NSIS  and  INSETs  are  specialized  units  within  the  RCMP),  RCMP  Bomb 
Squad,  Public  Health 

•  Outside  Agencies  -  Media 

•  Private  Sector  -  Salvation  Army 

3. 2. 3. 3  Scenario  3  -  infiuenza  Pandemic 

This  scenario  is  fictitious  but  based  on  the  recent  Severe  Acute  Respiratory  Syndrome  (SARS) 
outbreak  and  the  looming  threat  of  the  avian  flu  (H5N1)  virus.  A  variety  of  parties  would  need 
to  be  involved  in  order  to  set  up  and  enforce  quarantine,  conduct  widespread  health  monitoring 
and  testing,  and  take  steps  to  ensure  that  disease  vectors  are  bloeked. 

Organizations  that  could  be  involved  (as  well  as  DND): 

•  City/Provincial  Departments  -  Fire,  Police,  Ambulanee 

•  Federal  -  Health  Canada  (to  act  as  a  Federal  authority  on  this  health  matter,  to  involve 
other  appropriate  Federal  Ministries  (i.e.  Defence,  Finance,  Citizenship  and  Immigration 
etc.)  in  effecting  an  emergency  response).  Global  Public  Health  Intelligenee  Network 
(GPHIN)  Officials,  Centre  for  Emergency  Preparedness  and  Response  (CEPR)  (part  of 
Public  Health  Agency  of  Canada),  Pandemic  Influenza  Committee  (PIC) 

•  Outside  Ageneies  -  Media 

•  Private  Sector  -  Salvation  Army,  Red  Cross 
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Table  8:  Team  Research  Scenarios 


Points  of  interest 

Scenario:  Naturai  Disaster:  Winnipeg 

Fioods 

Scenario:  Terrorist  Threat 

Scenario:  Influenza  Pandemic 

Teams-of-teams 

Yes.  Local/provincial  officials  contact  higher 
authorities.  JTFPrairie  is  activated. 

CanadaCOM  and  JTFPrairie  work  with  Local 
and  Provincial  Authorities  such  as  the 

Manitoba  Emergency  Measures  Organization 
(EMO),  to  coordinate  the  disaster  response 
process,  plan  the  evacuation  of  danger  zones 
as  well  as  coordinate  resources  (water,  food 
and  shelter  to  sandbags  etc.). 

Yes.  Intelligence  of  a  threat  is  passed  on  to 
CanadaCOM  and  RCMP.  RCMP  is  tasked  as 
the  main  coordinator  for  evacuating  the  area 
and  ensuring  the  bomb  is  diffused  before  any 
damage  is  done.  Responsibilities  include  the 
investigation  of  any  offence  relating  to  a  threat 
to  the  security  of  Canada.  Joint  Task  Force 
Central  is  also  involved. 

Yes.  Local  BC  hospitals  have  seen  an  influx  in 
the  number  of  patients  being  treated  for 

Influenza.  The  Ministry  of  Health  is  contacted, 
and  subsequently  delegates  the  Public  Health 
Agency  of  Canada.  Assistance  is  requested 
from  CanadaCOM  to  help  coordinate  a  plan  of 
action  to  control  and  diffuse  the  pandemic. 

Clear  and  meaningful  start 
and  end  points 

Start:  Local  officials  are  unable  to  deal  with 
rising  flood  waters  and  associated  impacts, 
and  are  therefore  asking  for  assistance. 

End:  The  situation  is  managed,  natural 
escalation  (e.g.  rain,  snow  melt)  has  passed, 
dangers  are  minimized  and  local  authorities 
are  capable  of  dealing  with  the  situation. 

Start:  CanadaCOM  and  the  RCMP  have  been 
notified  of  intelligence  indicating  that  a  bomb 
will  explode  in  the  downtown  financial  district  of 
Toronto 

End:  Bomb  is  diffused,  and  citizens  are 
evacuated  in  a  timely  manner. 

Start:  Influenza  is  identified  and  confirmed  in 
multiple  human  cases  and  is  becoming  a 
pandemic  within  the  BC  region. 

End:  Influenza  vectors  are  known  and 
controlled  for. 

Joint 

Yes,  Land,  Sea  and  Air  may  be  involved. 

Yes,  the  Task  Force  (TF)  comprises  Land  and 

Air  Forces  (evacuate  citizens  using  helicopters 
and  other  aircraft). 

Yes,  the  TF  comprises  Land  and  Air  Forces. 

(The  Air  Force  can  be  used  to  evacuate 
civilians  and/or  bring  in  supplies) 

Interagency 

Yes,  there  are  agencies  from  DND  as  well  as 
OGDs  and  the  local  and  provincial  level. 

Yes,  there  are  agencies  from  the  Department 
of  National  Defence  (DND)  as  well  as  Other 
Government  Departments  (OGD) 

Yes,  there  are  agencies  from  the  Department 
of  National  Defence  as  well  as  Other 

Government  Departments  (OGD) 

Who  are  the  3  Teams? 

Team  1:  CanadaCOM 

Team  2:  Joint  Task  Force  Prairie 

Team  3:  Local/Provincial  Authorities  -  i.e. 
Ambulance,  Fire,  City  of  Winnipeg  Police 

Service 

Team  1:  CanadaCOM 

Team  2:  Joint  Task  Force  Central 

Team  3:  RCMP 

Team  1:  CanadaCOM 

Team  2:  Joint  Task  Force  Pacific 

Team  3:  Public  Health  Agency  of  Canada 
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Points  of  interest 

Scenario:  Naturai  Disaster:  Winnipeg 

Fioods 

Scenario:  Terrorist  Threat 

Scenario:  infiuenza  Pandemic 

Distributed 

Yes,  agencies  invoived  are  not  co-iocated. 

Yes,  agencies  invoived  are  not  co-iocated: 

Yes,  agencies  involved  are  not  co-iocated. 

CanadaCOM  -  Ottawa 

CanadaCOM  -  Ottawa 

CanadaCOM  -  Ottawa 

JTFPrairie-  Edmonton 

JTFCentrai- Toronto 

JTFPacific- Victoria 

Locai  Authorities  -  Winnipeg 

RCMP  -  nationai  security  reiated  criminai 
investigations  are  centraiiy  coordinated  by 
personnei  iocated  at  RCMP  Nationai 
Headquarters  (Ottawa). 

Public  Health  Agency  of  Canada  -  Winnipeg 

Who  are  the  team  members? 

Team  1:  CanadaCOM 

Team  1:  CanadaCOM 

Team  1:  CanadaCOM 

Higher  Commander 

Deputy  Commander 

Higher  Commander 

Deputy  Commander 

Higher  Commander 

Deputy  Commander 

Team  2:  Joint  Task  Force  Prairie 

Team  2:  Joint  Task  Force  Centrai 

Team  2:  Joint  Task  Force  Pacific 

Regionai  Commander  (and  2  of  the  3) 

J2  (Inteiiigence) 

J3  (Pians) 

J9  (CIMIC) 

Team  3:  Locai/Provinciai  Authorities 

Poiice, 

Fire 

Ambuiance 

Regionai  Commander  (and  2  of  the  3) 

J2  (Inteiiigence) 

J3  (Pians) 

J9  (CIMIC) 

Team  3:  RCMP 

Deputy  Commissioner  in  charge  of  the  Centrai 
region  (Ouebec  and  Ontario) 

Chief  Information  Officer 

Integrated  National  Security  Enforcement 

Teams  (INSETs)  (specialized  unit  within  the 
RCMP). 

Regional  Commander  (and  2  of  the  3) 

J2  (Intelligence) 

J3  (Plans) 

J9  (CIMIC) 

Team  3:  Public  Health  Agency  of  Canada 

Co-ordination  and  Operations  Group  (COG) 

Technical  Advisory  Group  (TAG) 

Emergency  Communications  Group  (ECG) 

How  are  teams  organized? 

Piease  refer  to  Figure  1 ;  more  detaii  regarding  organization  was  beyond  the  scope  of  this  contract. 
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Points  of  interest 

Scenario:  Naturai  Disaster:  Winnipeg 

Fioods 

Scenario:  Terrorist  Threat 

Scenario:  Influenza  Pandemic 

What  function  does  each  team 
perform?  (i.e.  strategic, 
operationai,  tacticai)? 

CanadaCOM  -  operationai 

JTFPrairie  -  operationai 

Locai  and  Provinciai  Authorities  - 

operationai 

CanadaCOM  -  operationai 

JTFCentrai  -  operationai 

RCMP  -  operationai 

CanadaCOM  -  operational 

JTFPacific  -  operational 

Public  Health  Agency  of  Canada  - 

operational 

Overaii  Goais 

Overaii  goai:  To  evacuate  and  accommodate 
aii  those  in  danger  and  minimize  property 
damage 

Without  conducting  a  detaiied  anaiysis  it  is  not 
possibie  to  specify  goais  and  priorities  at  the 
team  or  team  member  ievei.  This  ievei  of 
detaii  was  beyond  the  scope  of  this  contract. 

To  evacuate  area,  maintain  pubiic  cairn, 
manage  situation,  investigate,  ensure  other 
areas  are  not  under  threat,  protect  high  vaiued 
resources,  mobiiize  resources  surveiiiance, 
aiert  pubiic  and  necessary  agencies, 
debriefing,  reporting 

Without  conducting  a  detaiied  anaiysis  it  is  not 
possibie  to  specify  goais  and  priorities  at  the 
team  or  team  member  ievei.  This  ievei  of 
detaii  was  beyond  the  scope  of  this  contract. 

To  control  the  spread  of  the  pandemic,  to 
minimize  risk/threat,  use  of  precautionary 
measures,  preservation  of  health  within  the 
community,  to  maintain  order,  surveillance, 
vaccine  programs,  use  of  antivirals,  health 
services,  emergency  services,  public  health 
measures  and  communications. 

Without  conducting  a  detailed  analysis  it  is  not 
possible  to  specify  goals  and  priorities  at  the 
team  or  team  member  level.  This  level  of 
detail  was  beyond  the  scope  of  this  contract. 
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Points  of  interest 

Scenario:  Naturai  Disaster:  Winnipeg 

Fioods 

Scenario:  Terrorist  Threat 

Scenario:  infiuenza  Pandemic 

What  are  the  teams'  primary 
and  secondary  tasks? 

Team  1:  CanadaCOM  -  mobiiize  JTFPrairie, 
carry  out  strategic  goais 

Team  2:  JTFPrairie  -  Execute  the  miiitary 
support  required  by  the  iead  department  or 
ministry:  depioy  units  to  evacuate  peopie  in 
danger  zones,  ensure  open  iines  of 
communication  with  citizens,  accommodation 
for  those  evacuated,  protection  of  high  risk 
properties,  dike  construction,  sandbags, 
support  agencies  effectiveiy 

Team  3:  Locai  and  Provinciai  Authorities  - 

Coordinate  the  efforts  of  the  iocai  Poiice,  Fire 
and  Ambuiance  services,  as  weii  as  iocai 
pubiic  works.  Aiso,  use  iocai  knowiedge  to 
assist  other  agencies  (i.e.  DND)  to  be  most 
effective. 

Team  1:  CanadaCOM  -  mobiiize  JTFCentrai, 
carry  out  strategic  goais 

Team  2:  JTFCentrai  -  Depioy  units  to 
evacuate  peopie  in  danger  zones,  ensure  open 
iines  of  communication  with  citizens, 
accommodation  for  those  evacuated, 
protection  of  high  risk  areas,  support  agencies 
effectiveiy 

Team  3:  RCMP  -  the  investigation  of  any 
offence  reiating  to  a  threat  to  the  security  of 
Canada 

Team  1:  CanadaCOM  -  mobiiize  JTFPacific, 
carry  out  strategic  goais. 

Team  2:  JTFPacific  -  Depioy  units  to  assist  in 
controiiing  the  movements  of  the  popuiation, 
ensure  open  iines  of  communication  with 
citizens,  support  agencies  effectiveiy, 
contribute  to  the  overaii  surveiiiance  and  status 
of  the  situation,  ensure  that  the  heaith  of  CF 
personnei  and  citizens,  and  the  impact  on  CF 
operations  is  minimized 

Team  3:  Pubiic  Heaith  Agency  of  Canada  - 

tend  to  infected  and  deceased.  Provide  basic 
needs  to  citizens  (food,  water,  lodging, 
clothing)  hygiene,  public  health  (waste 
disposal,  assessment  of  vulnerable 
populations,  immunization,  psycho-social 
support,  quarantine  or  isolation,  identification  of 
deceased),  health  care  services  (triage) 

What  demands  are  imposed 
on  the  teams? 

Emergency  Situation: 

High  time  pressure 

High  ievei  of  uncertainty 

High  ievei  of  risk  to  pubiic  safety  (emergency 
crews  and  citizens) 

Emergency  Situation: 

High  time  pressure 

High  ievei  of  uncertainty 

High  ievei  of  risk  to  pubiic  safety  (staff  and 
bystanders) 

Emergency  Situation: 

High  time  pressure 

High  level  of  uncertainty 

High  level  of  risk  to  those  involved  in  trying  to 
control  the  pandemic  (staff) 

High  level  of  risk  to  public  safety 

Options  for  communication 
and  coordination 

Existing  Lines  of  Communications  (LOC)  such  as  iand  and  ceiiuiar  teiephone,  radio,  person  to  person,  pagers,  wireiess  toois  (internet,  instant 
messaging(chat)),  sateiiite  communications,  teieconferencing,  videoconferencing,  text-based  communication  (text  messaging  on  ceii  phones,  emaii), 
faxes 
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Points  of  interest 

Scenario:  Naturai  Disaster:  Winnipeg 

Fioods 

Scenario:  Terrorist  Threat 

Scenario:  infiuenza  Pandemic 

Highlight  key  decisions  and/or 
action  points  for  teams 

Team  1 :  CanadaCOM-  number  of  CF  to 
deploy,  allocate  resources,  specify  mission, 
specify  Command  and  Control  (C2) 
arrangements,  delegate  authority,  specify 
Transfer  of  Authority  (TOA) 

Team  2:  JTFPrairie  -  number  of  CF  to  deploy, 
allocate  resources,  specify  mission,  specify 
Command  and  Control  (C2)  arrangements, 
delegate  authority,  determine  Area  of 

Operations  (AOO),  direct  planning  for  the 
operation 

Team  3:  Local/Provincial  Authorities  - 

providing  overall  direction  and  coordinating 
activities  of  City  departments,  outside 
agencies,  the  public  sector  and  volunteer 
groups  during  an  emergency. 

Team  1:  CanadaCOM  -  number  of  CF  to 
deploy,  allocate  resources,  specify  mission, 
specify  Command  and  Control  (C2) 
arrangements,  delegate  authority,  specify 
Transfer  of  Authority  (TOA) 

Team  2:  JTFCentrai  -  number  of  CF  to 
deploy,  allocate  resources,  specify  mission, 
specify  Command  and  Control  (C2) 
arrangements,  delegate  authority,  determine 
Area  of  Operations  (AOO),  direct  planning  for 
the  operation 

Team  3:  RCMP  -  employ  investigate  methods 
and  tools  such  as  surveillance  and  the  use  of 
agents,  proper  handling  of  sensitive 
information 

Team  1 :  CanadaCOM  -  number  of  CF  to 
deploy,  allocate  resources,  specify  mission, 
specify  Command  and  Control  (C2) 
arrangements,  delegate  authority,  specify 
Transfer  of  Authority  (TOA) 

Team  2:  JTFPacific  -  number  of  CF  to  deploy, 
allocate  resources,  specify  mission,  specify 
Command  and  Control  (C2)  arrangements, 
delegate  authority,  determine  Area  of 

Operations  (AOO),  direct  planning  for  the 
operation 

Team  3:  Public  Health  Agency  of  Canada  - 

prioritization  of  needs,  alert  other  health 
institutions  (hospitals,  paramedics,  fire 
services),  determine  number  of  public  health 
staff  required,  what  basic  needs  are  required 
(amount  of  food,  water...),  amount  of 
medication  needed 

Wwaansy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


31 


HUMANSrSrt.US 


Points  of  interest 

Scenario:  Naturai  Disaster:  Winnipeg 

Fioods 

Scenario:  Terrorist  Threat 

Scenario:  infiuenza  Pandemic 

Define  positive  vs.  negative 
outcomes 

Positive:  aii  citizens  are  evacuated,  minimai 
casuaities,  minimai  property  damage, 
coordinate  maximum  number  of  resources  in 
shortest  amount  of  time 

Negative:  Mass  casuaities,  significant  property 
damage,  faiiure  to  coordinate  resources  in  a 
timeiy  manner 

Positive:  aii  citizens  are  evacuated  within 
danger  zone,  no  fataiities,  no  property  damage, 
coordinate  appropriate  resources  in  required 
amount  of  time,  safety  of  citizens  is 
maintained,  maintain  controi 

Negative:  Terrorist  threat  is  reaiised,  mass 
casuaities,  significant  property  damage,  faiiure 
to  coordinate  resources  in  a  timeiy  manner, 
incident  is  out  of  controi  (pubiic  chaos  and 
panic),  inappropriate  resources  depioyed  for 
threat 

Positive:  Transmission  rate  and  the  spread  of 
the  disease  is  significantiy  reduced,  the 
Pandemic  does  not  travei  beyond  the  BC 
border  or  into  new  area  within  BC,  minimai 
fataiities  and  infection,  coordinate  maximum 
number  of  resources  in  shortest  amount  of 
time,  safety  and  hygiene  of  citizens  is 
maintained,  maintain  controi 

Negative:  Influenza  had  spread  to  new  areas 
of  BC  and  may  also  have  crossed  beyond  the 

BC  borders  making  it  a  national  pandemic, 
mass  casualties  and  infection,  failure  to 
coordinate  resources  in  a  timely  manner, 
incident  is  out  of  control  -  citizens  are 
panicking,  rate  of  transmission  increases, 
safety  and  hygiene  of  citizens  is  at  jeopardy 

Identify  one  or  more  HSI 
intervention(s)  whose 
effectiveness  may  be  expiored 
using  these  scenarios 

Communication  toois  between/within  agencies 

Communication  channeis/media  for  pubiic  dissemination  of  info 

Decision  support  systems  for  command  teams 

Systems  to  enhance  situation  awareness  (a  necessary  precursor  to  decision  making),  notjust  of  the  evoiving  situation  and  how  it  wiii  evoive,  but 
aiso  of  the  possibie  resources  that  are  avaiiabie. 
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Points  of  interest 

Scenario:  Naturai  Disaster:  Winnipeg 

Fioods 

Scenario:  Terrorist  Threat 

Scenario:  infiuenza  Pandemic 

Further  develop  MOPs  and 
MOEs 

MOEs: 

Did  teams  accomplish  mission? 

Was  mission  accomplished  within  targets  (i.e.  budget,  time)? 

Did  all  teams  feel  they  were  used  effectively?  (possibly  a  subjective/questionnaire  measure) 

Number  of  lives  saved  (based  on  geographical  density  and  evacuation/quarantine  areas  developed) 

MOPs: 

Response  time  of  individual  teams  and  team  members' 

Number  of  resources  employed 

Match  of  resources  to  need 

Effectiveness  of  communications  between/within  teams 

Shared  team  mental  model(s) 

Quality  of  Plan 

Coordination  among  teams 

Ability  to  effectively  and  efficiently  use  all  the  resources  available  to  the  different  teams 

Individual  team  members'  workloads 

Number  of  outputs  made  in  specified  time 

Observed  versus  desired  outputs 

Prioritisation  of  tasks  and  information 

Which  factors  of  team 
performance  or  team 
effectiveness  can  each 
scenario  explore? 

Shared  knowledge,  communication,  coordination,  team  adaptability,  planning,  task  complexity,  workload 

Which  factors  are  expected  to 
be  especially  influential  in  the 
scenarios? 

Time  Pressure,  Task  Complexity,  Task  Interdependence,  Planning,  Individual  Experience,  Stress,  Risk,  Availability  of  (new)  Information, 

Prioritisation,  Participant  Availability 

For  each  influential  factor, 
what  level  of  the  factor  does 
this  scenario  represent? 

Each  factor  can  be  manipulated  to  suit  the  experimental  aims  of  the  research 
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Points  of  interest 

Scenario:  Naturai  Disaster:  Winnipeg 

Fioods 

Scenario:  Terrorist  Threat 

Scenario:  infiuenza  Pandemic 

How  well  do  scenarios  support 
team  experiments? 

How  well  do  scenarios  support 
computational  modelling  of 
teams? 

All  scenarios  are  flexible  enough  to  accommodate  Investigations  Into  the  relevant  team  factors,  processes,  etc.  Identified  In  Sartorl  et  al  (2006). 

Scenario  can  provide  Inputs,  entitles  Involved,  processes.  Individual  tasks  and  outputs. 

The  three  team  scenarios  developed  for  this  report  are  meant  to  serve  as  a  starting  point  for  developing  a  customized  domestic  scenario  that  can  be 
used  for  either  live  team  experiments  or  computational  modelling  of  teams.  However,  before  this  can  be  done,  the  following  types  of  Information 
should  be  further  explored: 

•  Network  configuration  between  teams  and  within  teams  (I.e.  centralized  network,  decentralized  network  or  hierarchical  network). 

•  Similarities  and  difference  In  team  tasks/goals  and  team  members'  tasks/goals. 

Is  there  a  conceptual  model  of 
team 

performance/effectiveness 
(from  literature)  that  can  be 
tested  by  experiments  using 
this  scenario? 

1)  Command  Team  Effectiveness  Model:  Essens  et  al.  (2005):  model  of  team  performance  that  was  developed  In  order  to  Identify  critical  factors  In 
command  team  effectiveness 

2)  Contextual  Model  of  Groupware  Development:  Driskell  &  Salas  (2006)  contextual  model  of  groupware  development 

Is  there  a  computational 
model  (I.e.  IPME)  of  team 
performance/effectiveness 
(from  survey)  that  can  be 
tested  by  experiments  using 
this  scenario? 

No:  existing  computational  models  are  for  Intact  teams  performing  tasks  at  a  tactical  level 

To  conduct  experiments  with 
low/medlum  fidelity  and 
medlum/high  control,  who 
should  participate  In  the 
experiment  (I.e.  profile 
sketch)? 

Participants  should  have  operational  experience,  be  familiar  with  relevant  policies  and  procedures  for  emergency  planning  and  response 

Commanders  who  are  fully  conversant  with  the  tactics,  techniques,  capabilities,  needs  and  limitations  offerees,  and  the  environment. 

However,  with  adequate  briefing  materials,  time  to  familiarise  and  a  carefully  managed  and  bounded  scenario  (communicated  by  the  briefing 
materials)  novices  could  be  drafted  In  to  participate.  This  would  be  problematic  though,  because  each  of  the  three  teams  would  therefore  be  an  ad 
hoc  team,  decreasing  the  control  over  the  experiment. 

What  screening/training 
should  be  provided  to  the 
experiment  participants? 

Training  -  Doctrine  for  emergency  planning,  relevant  software  and  hardware  that  will  be  used  to  conduct  experiment,  adequate  briefing  of  roles, 
responsibilities,  available  Information,  available  resources,  etc.  (If  option  to  use  novices  Is  pursued). 

Screening  -  Relevant  operational  experience,  familiarization  with  emergency  planning  and  response 
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As  mentioned  in  Table  8,  the  most  relevant  coneeptual  models  to  the  team  research  scenarios  are 
Essens  et  al.  (2005)  and  Driskell  &  Salas  (2006).  Essens  et  al  (2006)  present  the  Command  Team 
Effectiveness  Model  (CTEM)  to  identify  critical  team  performance  factors  in  command  team 
effectiveness.  The  Command  Team  Effectiveness  Model  (CTEM)  is  illustrated  in  Eigure  4. 
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Figure  4:  Command  Team  Effectiveness  Model  (Essens  et  al.  (2005)) 

As  illustrated  in  the  figure  above,  many  factors  presented  in  the  CTEM  model  are  also  found  in 
the  criteria  that  came  from  the  literature  survey  including  uncertainty,  complexity,  workload, 
team  size,  planning,  etc.  The  other  model  that  may  be  applicable  to  our  scenarios  is  the 
Contextual  Model  of  Groupware  Development  developed  by  Driskell  &  Salas  (2006)  (Eigure  5). 
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Figure  5:  Contextual  Model  of  Groupware  Development  (Driskell  &  Salas  (2006)) 

This  model  supports  distributed  teams  and  is  geared  towards  team  functions  that  support  team 
performance.  Therefore  a  large  emphasis  is  placed  on  team  processes.  More  detailed 
explanations  of  both  these  models  can  be  found  in  Sartor i  et  al.  (2006).  The  primary  reason 
these  conceptual  models  are  deemed  most  relevant  to  the  team  research  scenarios  developed  in 
this  project  is  the  scenarios  that  were  generated  include  a  complex  number  of  tasks,  factors  and 
processes,  and  therefore  most  closely  matched  conceptual  models  that  adequately  represented  the 
interplay  of  multiple  tasks,  factors  and  processes.  Other  models  reviewed  by  Sartori  et  al  (2006) 
are  less  inclusive  and  address  a  smaller  range  of  factors,  ignoring  the  role  of  contextual  or 
situational  factors.  Conversely,  both  Essens  et  al.  (2005)  and  Driskell  &  Salas  (2006)  provide 
conceptual  models  that  perform  more  than  a  limited  aspect  of  team  performance,  and  though  not 
corresponding  precisely  to  the  scenarios  developed  for  this  project,  these  models  are  the  better 
conceptual  matches. 

3.3  Mapping  the  developed  scenarios  to  the  criteria 

Earlier,  the  criteria  identified  in  the  literature  review  (Sartori  et  al.,  2006)  were  mapped  to  the 
reviewed  scenarios.  To  get  a  sense  of  how  the  developed  scenarios  compare  to  the  reviewed 
scenarios,  the  developed  scenarios  have  been  mapped  according  to  the  same  criteria  that  came 
out  of  the  literature  review. 

As  noted  earlier,  team  factors  consist  of  criteria  such  as  team  size,  team  history  and  physical 
distribution.  Most  reviewed  scenarios  used  teams  that  were  small  in  size.  Conversely,  the 
developed  scenarios  would  include  three  teams  of  2-3  members  each.  Two  of  the  teams  would 
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be  of  DND  (CanadaCom  and  JTF)  origin  and  the  third  team  would  be  an  OGD.  Each  sub  team 
is  considered  small  while  the  entire  teams-of-teams  structure  is  of  medium  size  (N  =  8) .  The 
majority  of  the  reviewed  scenarios  focused  on  fixed  teams.  The  developed  scenarios  provide 
opportunities  for  both  fixed  and  ad  hoc  teams.  The  individual  teams  (CanadaCom,  JTF,  OGD) 
are  expected  to  be  fixed  as  individuals  within  teams  likely  have  a  history  of  working  together. 
CanadaCom  and  the  JTF  are  also  likely  to  be  a  fixed  team  since  they  are  expected  to  have 
worked  together  in  the  past.  Even  if  these  two  teams  have  not  had  the  opportunity  to  work 
together,  it  is  likely  that  DND  has  in  place  set  procedures  governing  responsibilities  and 
interactions.  On  the  other  hand.  The  DND  teams  and  the  OGD  team  are  more  likely  to  be  an  ad 
hoc  team.  Although  it  is  conceivable  that  DND  has  worked  with  OGDs,  it  may  not  be  the  case 
that  the  specific  OGD  involved  in  the  scenario  has  worked  with  DND  in  the  past.  The  reviewed 
scenarios  used  both  distributed  and  co-located  teams.  The  developed  scenarios  propose  to  do  the 
same  thing.  The  individuals  within  each  team  are  co-located  while  the  headquarters  each  of  each 
team  are  distributed  across  the  country;  therefore  sub  teams  are  co-located  while  teams-of-teams 
are  distributed. 

Task  factors  consist  of  criteria  such  as  task  complexity,  workload  and  task  interdependence. 

More  than  half  of  the  reviewed  scenarios  emphasized  uncertainty  as  an  important  variable.  The 
developed  scenarios  can  accommodate  varying  levels  of  uncertainty  since  the  nature  of  the 
scenarios  is  an  emergency  situation  where  teams  are  required  to  manage  a  dynamic  and 
constantly  changing  set  of  circumstances.  In  terms  of  workload,  reviewed  scenarios  focused  on 
cognitive  work.  The  developed  scenarios  also  focused  on  cognitive  workload  as  teams  are 
actively  involved  in  decision  making,  planning  etc.  To  increase  the  impact  of  the  emergency 
situation,  experimenters  can  manipulate  time  pressure.  In  the  Winnipeg  floods  scenario, 
experimenters  can  increase  the  rate  at  which  flood  waters  rise;  in  the  terrorist  threat  scenario 
teams  could  be  alerted  of  intelligence  regarding  another  bomb  in  a  different  location;  in  the 
influenza  pandemic  scenario  the  rate  of  transmission  could  occur  at  an  unprecedented  rate. 
Among  reviewed  scenarios,  additive  tasks  were  most  common.  The  developed  scenarios  could 
accommodate  additive  type  tasks,  as  well  as  other  types  of  tasks.  In  each  of  the  developed 
scenarios,  many  tasks  are  being  performed  by  individuals  with  different  roles.  The  extent  and 
type  of  interdependence  within  teams  and  between  teams  is  unclear.  However,  the  magnitude  of 
the  scenarios  makes  it  unlikely  that  teams  could  succeed  if  they  worked  in  isolation. 

Team  processes  consist  of  criteria  such  as  shared  knowledge,  communication,  coordination,  team 
adaptability  and  planning.  In  the  developed  scenarios,  mental  models  could  be  examined  on 
various  levels.  Team  mental  models  may  refer  to  the  resulting  mental  model  of  each  team  or  the 
teams-of-teams  mental  model.  Task  mental  models  could  be  referring  to  the  tasks  given  to  each 
team  member,  the  tasks  given  to  each  team,  or  the  task  given  to  all  teams  as  a  whole.  Similarly, 
implicit  and  explicit  communication  as  well  as  implicit  and  explicit  coordination  can  occur  at 
various  levels  -  between  individuals  and  between  teams.  It  is  expected  that  in  the  developed 
scenarios,  the  network  configuration  within  teams  will  be  a  hierarchical  network,  especially 
within  the  DND  teams.  This  is  because  the  CF  is  hierarchical  by  nature  and  the  developed 
scenarios  aim  to  accurately  depict  real  team  configurations.  What  is  unclear  for  each  of  the 
scenarios  is  what  the  teams-of-teams  network  configurations  would  be.  Furthermore,  if  teams- 
of-teams  formed  a  centralized  network  it  would  be  necessary  to  identify  the  lead  team.  In  the 
developed  scenarios,  team  adaptability  is  important  to  monitor,  especially  if  planned  modifying 
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inputs  (discussed  in  section  3.2.2)  are  integrated  into  the  scenarios.  How  well  teams  are  able  to 
respond  to  changes  and  correct  for  inefficient  practices  can  provide  valuable  insights.  Resource 
allocation  can  also  be  triggered  by  planned  distracter  inputs  (discussed  in  section  3.2.2)  which 
require  team  members  to  manage  non-emergency  related  events  that  may  arise  during  the 
emergency  scenarios.  Research  into  team  processes  that  are  relevant  to  the  CF  can  lead  to  the 
identification  of  hardware  and  software  requirements  (such  as  video  teleconferencing,  wireless 
devices  etc.)  that  can  enhance  team  capabilities. 

Most  reviewed  scenarios  measured  performance  automatically.  Although  not  specified  by  the 
developed  team  research  scenarios,  computers  could  measure  a  variety  of  MOPs  such  as  the 
number  and  types  of  available  resources  in  relation  to  the  number  and  types  of  resources 
employed,  number  of  outputs  made  etc.  Self  reports  could  be  employed  to  measure  MOPs  such 
as  the  effectiveness  of  communication  between/within  teams  and  perceived  workload.  Observers 
could  supplement  these  measures  by  evaluating  subjective  points  such  as  the  quality  of  the  plan, 
and  to  what  extent  did  teams  accomplish  their  mission.  In  addition,  the  levels  of  analysis  for  the 
team  research  scenarios  are  at  both  the  individual  and  team  levels  as  the  mission  requires  the 
accomplishment  of  group  and  individual  goals. 

Comparing  reviewed  scenarios  to  those  produced  specifically  for  this  project  highlights  how  the 
developed  scenarios  differ.  Specifically,  for  the  natural  disaster,  pandemic  and  terrorist  threat 
scenarios  the  team  factors  are  somewhat  different  from  the  teams  used  in  the  reviewed  scenarios. 
The  task  factors  highlighted  in  the  reviewed  scenarios  are  similar  to  the  task  factors  highlighted 
by  the  developed  scenarios.  Without  actually  specifying  in  more  detail  the  team  research 
scenarios,  it  is  difficult  to  assume  which  processes  would  be  the  result  of  team  interactions. 
However,  as  the  team  research  scenarios  stand,  they  are  capable  of  supporting  all  team  processes 
from  shared  knowledge,  to  communication,  planning,  team  adaptability  and  coordination. 

Lastly,  which  performance  measures  to  use  and  how  to  collect  them  will  depend  greatly  on  the 
details  of  the  experiment,  but  in  its  present  state,  the  team  research  scenarios  could  employ 
automated,  self-report  and  observer  measures. 
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4.  Results  of  Evaluation  of  Computational 
Modelling  Tools/Approaches 

IPME  is  a  potential  tool  for  developing  a  computational  model  of  the  targeted  teams  in  the 
targeted  context,  but  it  may  or  may  not  be  the  most  appropriate  tool.  This  evaluation  attempted  to 
conclude  whether  IPME  can  be  used,  with  reasonable  effort,  to: 

model  the  specified  team  tasks? 

model  the  specified  team  interactions? 

model  the  various  HSI  interventions  that  may  be  of  interest? 

analyze  the  team’s  strategies? 

analyze  the  team’s  performance? 

In  addition,  this  evaluation  assessed  if  and  how  IPME  can  be  used  to  model  the  specified  team  as 
an  entity,  as  well  as  if  and  how  it  can  be  used  to  model  the  team  as  a  collection  of  individuals. 
With  respect  to  the  possible  HSI  interventions,  this  evaluation  should  consider  whether  a  whole 
new  IPME  model  would  need  to  be  developed  for  each  new  intervention  or  each  new  level  of  an 
existing  intervention,  or  whether  interventions  can  be  modelled  as  modules  that  are  added  to  or 
removed  from  the  main  model  as  needed.  If  appropriate,  enhancements  or  alternatives  to  IPME 
should  be  proposed. 

The  goal  of  this  project  is  to  research  and  assess  various  computational  models  based  on  a  series 
of  assessment  criteria  in  terms  of  team  processes  modelling.  A  total  of  26  platforms  have  been 
evaluated  in  this  task  according  to  14  criteria  (see  Table  2). 

Several  key  concepts  are  explained  as  follows: 

Constructive  Simulations:  This  term  is  relative  to  live  or  virtual  simulations  both  of  which 
involve  real  people  operating  real  or  simulated  systems  respectively.  In  contrast,  constructive 
simulations  are  simulations  that  involve  simulated  people  operating  simulated  systems.  Real 
people  stimulate  (make  inputs  to)  such  simulations,  but  are  not  involved  in  determining  the 
outcomes.  Computational  models  of  human  behaviour  potentially  provide  the  simulated  people 
for  the  constructive  simulations,  i.e.  synthetic  forces  for  constructive  military  simulations  (for 
example,  in  the  integration  of  CAST  and  DDD,  the  CAST  architecture  was  extended  to  replace 
some  or  all  of  the  players  in  a  DDD  simulation  task).  Generally  speaking,  all  the  models 
identified  in  this  report  are  constructive  simulations,  except  for  Wildfires  Fight  Simulation  for 
Training. 

Computational  Cognitive  Models:  Computational  cognitive  models  are  integrated  models  of 
how  humans  perform  complex  cognitive  tasks,  e.g.  human  cognition,  perception,  sensation, 
motor  action  and  knowledge,  that  embody  a  principled  underlying  theory  or  framework  for 
human  information  processing.  These  models,  which  can  be  run  on  a  computer,  capture  human 
knowledge  in  an  abstract  form  and  allow  behaviour  and  cognition  to  be  simulated  across  a  broad 
range  of  situations.  Such  models  can  provide  a  priori  performance  predictions  of  how  well  a 
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certain  system  will  support  the  tasks  workers  perform  by  assessing  factors  such  as  how  easy  the 
system  will  be  to  learn  and  use,  the  workload  it  imposes,  and  the  propensity  for  errors.  Software 
agents  that  perform  work  tasks  in  the  same  way  that  humans  perform  work  tasks  can  be  used  to 
evaluate  proposed  system  designs  without  the  need  to  conduct  these  types  of  evaluations  with 
aetual  workers.  This  class  of  models  include  ACT-R,  COGNET,  EPIC,  and  SOAR,  among  others. 

Computational  Task  Network  Models  :  These  models  are  the  analogue  of  computational 
cognitive  models,  but  instead  focus  only  on  modelling  the  overt  behaviours  necessary  to  perform 
tasks,  rather  than  the  underlying  cognitive  activities  that  drive  task  performanee.  Typieally, 
human  performance  data  that  have  been  previously  collected  are  provided  as  input  to  the 
simulation.  The  simulation  ean  either  simulate  graphically  the  environment  and  workspaee,  or 
dynamieally  "run"  the  task  in  real  or  fast  time  as  a  way  of  estimating  complete  cycle  times,  error 
likelihoods,  workload,  etc.  These  techniques  can  be  used  to  assess  potential  contributions  of 
alternative  configurations  of  tasks,  equipment,  and  team  organizations.  They  can  also  aid  in  the 
design  and  analysis  of  tasks  by  assessing  how  the  characteristics,  interactions,  and  sequences  of 
tasks  ean  impact  operator  workload.  Further,  they  can  be  used  to  assess  the  effeets  of  proposed 
changes  to  an  existing  system  on  operator  workload  and  productivity  without  the  need  for  person- 
in-the-loop  testing.  This  class  of  models  includes  IPME  and  IMPRINT. 

Multi-Agent  Models:  There  has  been  growing  interest  in  using  intelligent  agents  to  model  and 
simulate  human  teamwork  behaviours.  The  five  multi-agent  teamwork  simulation  tools  reviewed 
during  the  course  of  this  research  are  CAST,  Brahms,  COGNET /BATON,  STEAM  and  Team- 
SOAR.  These  models  were  specially  designed  to  represent  aspects  of  teamwork  that  are  not  found 
in  most  models  (e.g.,  collaboration,  “off-task”  behaviours,  multitasking,  interrupt  and  resume, 
informal  interaction,  and  geography).  This  suggests  that  this  elass  of  models  would  be  highly 
applicable  to  military  scenarios  requiring  collective  action,  such  as  tactical  planning  and 
preparing.  It  would  seem  that  these  multi-agent  models,  which  are  strong  on  social  interaction 
but  weak  on  individual  cognition,  would  make  a  perfect  match  with  Soar  or  ACT-R,  which  are 
strong  on  individual  cognition  but  weak  on  social  interaction.  COGNET /BATON,  STEAM  and 
Team-Soar  are  examples  of  efforts  towards  this  direction. 

4.1  Results 

Annex  F  compares  the  computational  models  with  regard  to  the  team  process  functions  they 
simulate.  The  table  entries  generally  describe  the  outcome  of  dichotomous  yes/no  judgments  of 
whether  each  model  is  capable  of  emulating  the  function  in  question.  Whenever  neeessary  or 
available,  a  short  textual  passage  is  given  to  describe  model  capability  with  regard  to  the  function 
as  sort  of  rationale  for  the  ratings  assigned  to  the  evaluation  criteria  for  eaeh  team  modeling 
platform.  Although  the  research  is  extensive,  it  is  by  no  means  an  exhaustive  list  of 
eomputational  models.  In  the  meantime,  the  judgements  of  each  criterion  were  based  on  the  three 
methods  described  in  the  Approach  section  and  documents  listed  in  the  References  section. 
References  that  were  not  unearthed  may  eontain  evidenee  for  additional  model  eapabilities.  The 
resultant  judgements  are  not  conclusive  but  tentative  and  suggestive,  considering  the  limited  time 
and  resourees  spent  on  the  project  and  the  complexity  and  flexibility  of  the  application  of 
computational  models.  Indeed,  even  domain  experts  of  certain  computational  models  have 
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different  judgements  on  the  assessment  criteria.  References  are  provided  for  some  applications  in 
Annex  F.  They  serve  as  information  for  further  research  on  the  issues. 

Given  the  present  format,  a  few  words  of  caution  are  appropriate  in  interpreting  Annex  F: 

Although  every  effort  was  made  to  qualify  each  yes/no  judgment  with  a  description  as  to  the 
quality  and  the  extent  to  which  the  model  actually  models  a  particular  function,  the  reader  may 
find  that  many  cells  are  filled  simply  with  yes/no  answers;  these  cells  are  either  self-explanatory 
or  there  was  limited  available  relevant  information.  In  those  cases,  the  reader  should  regard  the 
entries  as  suggestive  and  consult  the  appropriate  references  for  a  more  detailed  description  of 
model  capabilities. 

To  earn  a  “yes”  judgment,  either  the  documentation  and  other  literature  or  the  responses  from 
the  questionnaires  or  interviews  associated  with  the  model  had  to  indicate  or  describe  the  model’s 
capabilities  specifically  for  that  particular  function.  Sometimes  inferences  (educated  guesses) 
were  made  about  the  model’s  potential  capabilities  in  order  to  fill  in  the  cell. 

Despite  appearances.  Annex  F  does  not  represent  a  “scorecard”  with  which  to  rate  the  merits  of 
computational  models.  The  fact  that  one  model  emulates  5  functions  and  another  emulates  10 
functions  does  not  reflect  their  relative  worth  to  model  users  (see  Table  9  below).  The  match  of 
model  functions  to  the  simulation  requirement  is  what  should  matter  to  the  user. 
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Table  9:  Number  of  Criteria  Met  by  each  Application 


Name  of 
Application 

Measure 

workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model 

Team 

as 

Entity 

Model 
Team  as 
Group  of 
Individuals 

Model 

Specified 

Team 

Tasks 

Model 

Specified 

Team 

Interaction 

Model  HSI 
Intervention 
of  Interest 

Analyze 

Team's 

Strategies 

Analyze 

Team's 

Performan 

ce 

Available 
in  Public 
Domain 
(i.e.  free) 

stable 

Real-time 

Computer 

Generated 

Forces 

Number  of 
Criteria 
Matches 

IPME 

V 

N/A 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

12 

IMPRINT 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

13 

BRAHMS 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

13 

SOAR 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

13 

MIDAS 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

12 

STEAM 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

12 

D-OMAR 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

11 

DDD 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

11 

TOD 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

11 

Archimedes 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

11 

JIMM 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

11 

CAST 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

10 

C3TRACE 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

10 

RESA 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

10 

SAMPLE 

V 

V 

V 

V 

V 

V 

V 

V 

V 

9 

GLEAN 

V 

V 

V 

V 

V 

V 

V 

V 

V 

9 

APEX 

V 

V 

V 

V 

V 

V 

V 

V 

V 

9 

EADSIM 

V 

V 

V 

V 

V 

V 

V 

V 

8 

COGNET 

V 

V 

V 

V 

V 

V 

V 

V 

8 

ACT-R/PM 

V 

V 

V 

V 

V 

V 

6 

PUMA 

V 

V 

V 

V 

V 

5 

A-SA 

V 

V 

V 

V 

4 

KOGSIT 

V 

V 

V 

V 

4 

EPIC 

V 

V 

V 

V 

4 

Cogitoid 

V 

V 

V 

3 

Wildfires 

V 

V 

V 

3 
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This  report  describes  the  output  of  two  streams  of  work.  The  first  objective  was  to  develop 
scenarios  that  could  be  used  for  the  purposes  of  team  research.  This  was  done  by  leveraging 
previous  work  (Sartori  et  al.,  2006)  conducted  for  this  ARP.  The  second  objective  was  to  review 
available  computational  modelling  applications  and  determine  which  model  would  be  most 
appropriate  for  modelling  team  research  scenarios  in  order  to  provide  insights  into  teams,  team 
processes,  team  factors  and  human-systems  integration  interventions.  As  an  additional  point,  the 
resulting  computational  model  needed  to  support  modelling  for  the  team  scenarios  developed  for 
this  project.  This  section  considers  general  observations  about  the  developed  scenarios  and  the 
computational  models  assessed  and  their  implications. 

5.1  Scenario  Development 

Three  scenarios  were  developed,  a  natural  disaster  scenario,  a  terrorist  threat  scenario  and  a 
pandemic  scenario.  These  scenarios  were  developed  with  multiple  research  themes  in  mind  (i.e. 
team  factors,  task  factors,  team  processes  and  measures  of  performance),  with  practical 
application  to  organizations  that  could  benefit  from  team  research  such  as  CanadaCOM,  Joint 
Task  Forces,  PSEPC,  etc.  Therefore,  the  scenarios  generated  are  geared  toward  CF 
requirements  in  support  of  unexplored  areas  of  team  research.  Further,  the  three  scenarios  were 
created  as  an  appropriate  context  for  studying  HSI  interventions  and  to  serve  as  a  basis  for  the 
computational  modelling  of  teams. 

5.1.1  Implications  for  team  research 

This  report  identifies  aspects  of  team  performance  that  have  been  investigated  using  previous 
scenarios  while  highlighting  less  researched  and  unexplored  areas  of  team  research.  In  using  this 
approach  it  was  hoped  that  the  scenarios  designed  could  help  breach  gaps  in  knowledge  about 
teams,  while  drawing  on  the  successes  of  previous  experiments.  As  a  result,  two  main  themes 
relevant  to  team  research  arose  -  investigation  into  less  researched  areas,  such  as  team  factors, 
and  the  development  of  scenarios  that  support  the  investigation  of  complex  interactive,  multi¬ 
factor,  models  instead  of  those  that  support  the  investigation  of  a  single  criterion. 

Previous  team  research  has  focused  on  teams  with  the  following  characteristics  -  small  sized 
teams,  fixed  teams,  and  co-located  or  distributed  teams.  The  scenarios  developed  for  this  project 
propose  to  explore  the  multiple  side  of  the  team  factors  continuum  by  using  medium,  teams-of- 
teams  that  are  both  ad  hoc  and  fixed,  and  distributed  and  co-located.  Since  limited  attention  has 
been  given  to  some  of  these  factors,  the  scenarios  were  developed  to  target  this  gap  with  the  aim 
of  offering  empirical  insight  into  an  unexplored  area  of  team  research  (such  as  team  processes 
arising  out  of  these  specific  types  of  teams). 

By  initiating  a  new  stream  of  research,  that  is  particularly  applicable  to  real  life  teams  (i.e.  CF 
involvement  in  domestic  emergency  situations),  it  is  hoped  that  insightful  and  applicable 
outcomes  will  result.  Further  investigation  into  team  factors,  for  example,  could  allow 
researchers  to  draw  conclusions  on  optimal  composition  and  structure  of  teams  that  is  relevant  to 
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the  type  of  task  being  performed.  Likewise,  understanding  the  mediating  factors  in  shared 
knowledge  could  lead  to  the  development  of  tools  and  techniques  to  enhance  shared  knowledge. 
As  a  result  of  these  insights,  DND  could  formulate  teams  to  maximize  team  performance  on  the 
basis  of  this  research. 

The  scalability  of  the  scenarios  also  lend  themselves  to  the  development  of  team  research.  The 
scenarios  presented  allow  for  the  manipulation  of  a  variety  of  factors  relevant  to  team 
performance.  For  example,  with  the  terrorist  threat  scenario  different  aspects  of  team 
performance  can  be  targeted  such  as  uncertainty,  shared  knowledge,  time  pressure, 
communication,  etc.  All  scenarios  are  capable  of  dealing  with  more  than  a  single  factor  in 
relation  to  team  performance  because  the  scenarios  are  capable  of  supporting  complex  and 
interactive  processes.  For  example,  the  scenarios  can  support  the  interactive  effects  of  shared 
knowledge  and  communication  on  team  performance.  It  is  hoped  that  using  a  complex  scenario 
will  prove  fruitful  in  representing  the  different  demands  imposed  on  a  real  life  team.  Although 
as  the  complexity  of  a  scenario  increases,  and  the  number  of  interactive  processes  increase,  there 
is  the  danger  of  reduced  experimental  control.  It  is  therefore  necessary  that  the  specific  objective 
of  any  experiment  involving  these  new  scenarios  be  clearly  defined  in  order  to  enable  the 
research  team  to  strike  a  balance  between  experimental  control,  fidelity  and  validity. 

5.1.2  Implications  for  HSI  interventions 

The  developed  scenarios  offer  an  appropriate  context  for  conducting  human  systems  integration 
research  on  teams,  their  processes,  tools  and  tasks.  Due  to  the  complexity  of  the  developed 
scenarios,  they  provide  a  broad  spectrum  of  capabilities  and  therefore  provide  the  opportunity  to 
explore  relationships  within  teams,  between  teams,  and  with  technology  in  a  complex  setting. 

The  purpose  of  determining  and  creating  appropriate  HSI  interventions  is  to  help  the  team  and 
system  realize  the  required  level  of  performance.  Performance  in  this  case  can  be  measured  at 
several  levels,  from  how  well  the  task  was  performed,  how  well  the  teams-of-teams  collectively 
performed,  how  well  each  individual  team  performed,  to  how  well  each  individual  team  member 
performed. 

The  scenarios  developed  for  this  project  also  support  such  traditional  HSI  methods  as  Mission 
Function  Task  Analysis  (MFTA).  MFTA  conducted  on  the  experimental  scenarios  would  serve 
as  a  baseline  understanding  of  what  the  scenario  is  supposed  to  achieve.  This  understanding 
would  allow  objective  evaluation  throughout  the  introduction  of  new  and  varied  HSI 
interventions,  such  that  the  researchers  would  be  certain  that  any  measured  change  in  scenario 
performance  would  be  attributable  to  the  intervention  and  not  random  differences.  For  example, 
MFTA  may  show  that  the  critical  function  for  one  of  the  scenarios  is  the  allocation  of  appropriate 
levels  of  personnel  to  strategic  points  (e.g.  in  time  or  space)  to  curtail  the  spread  of  the  threat. 
This  requirement  (to  have  a  baseline  understanding)  remains  constant  irrespective  of  the  research 
aims  or  the  HSI  intervention  being  assessed.  If  a  decision  support  tool  is  implemented,  the 
function  remains  the  same  and  the  researchers  look  for  changes  in  performance.  If  a  new 
organisational  structure  is  implemented,  again,  the  mission  stays  the  same  and  researchers  look 
for  changes  in  performance.  The  compatibility  of  these  scenarios  with  HSI  leverages  a  great  deal 
of  experience  and  understanding  from  throughout  the  CF  and  beyond. 
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5.1.3  Implications  for  the  CF 

In  addition  to  developing  scenarios  to  make  strides  within  the  area  of  team  research,  the 
scenarios  have  been  developed  to  address  the  future  needs  of  the  CF.  The  context  within  which 
the  CF  operates  is  changing  and  as  a  result  new  issues  and  skills  have  become  significant.  The 
CF  may  face  more  complex  threats  in  the  future,  and  therefore  threat  response  requires  greater 
collaboration  between  different  levels  of  government  and  different  organizations.  The  CF  must 
be  prepared  to  deal  with  this  increase  in  scope  and  complexity  in  order  to  provide  the  level  of 
stability  and  security  demanded  by  the  Canadian  people.  In  addition,  many  organizations  exist 
beyond  the  boundaries  of  the  CF  who  are  capable  of  contributing  resources  to  deal  with  modern 
day  events.  It  will  therefore  become  more  common  for  the  CF  to  combine  resources  and 
capabilities  with  other  government  departments  or  non-government  organizations  to  effectively 
deal  with  emergency  events. 

As  the  number  of  teams  and  players  increase,  new  issues  impact  the  resolution  of  a  situation. 
Individuals  from  different  organizations,  with  different  backgrounds  and  training  must  work 
collectively  to  accomplish  a  task.  Teams  must  share  and  divide  authority  and  responsibility, 
ensuring  that  all  available  resources  are  appropriately  used.  Teams  must  be  capable  of 
organizing  and  contributing  to  a  mission  from  different  physical  locations  and  must  do  so  within 
a  short  timeline,  with  minimal  time  to  prepare.  All  these  issues  must  be  managed  so  that  mission 
accomplishment  can  remain  the  focal  point. 

The  developed  scenarios  have  therefore  been  created  with  the  following  teams  in  mind: 
interagency,  joint,  ad  hoc,  and  distributed,  teams-of-teams.  For  example,  the  Winnipeg  Floods 
Natural  Disaster  Scenario  draws  on  resources  from  various  teams.  Three  main  teams 
representing  different  organizations  are  employed  -  CanadaCOM,  Joint  Task  Force  Prairie  (land 
and  air  forces),  and  Local  and  Provincial  authorities.  Incorporating  these  types  of  teams  into  a 
scenario  will  impose  the  constraints  and  limitations  similar  to  what  the  CF  is  expected  to  face  in 
the  near  future.  By  acknowledging  and  recognizing  the  changing  face  of  teams  involved  in 
emergency  situations,  the  CF  and  other  organizations  can  begin  training  and  planning  so  that 
team  performance  is  maximised  from  the  onset  of  the  teams  formation. 

Research  into  teams  will  benefit  the  CF  by  providing  a  foundation  on  which  to  build  their  teams. 
The  CF  will  gain  better  insights  into  how  to  organize  teams  for  different  types  of  tasks  and  what 
processes  emerge  out  of  team  interactions.  The  CF  can  therefore  build  supporting  structures 
such  as  communication  networks,  training,  facilities,  etc,  around  teams,  rather  than  teams  around 
structures.  For  example,  if  research  indicates  that  shared  team  knowledge  is  facilitated  for 
distributed  teams  through  video  teleconferencing,  than  the  CF  can  incorporate  this  technology 
into  future  structures  and  training. 

5.1.4  Implications  for  computational  models 

The  computational  model  chosen  for  the  purpose  of  this  ARP  should  be  able  to  represent  the 
chosen  scenario.  However,  the  limits  of  the  modelling  tool  may  be  met  as  the  scenarios  become 
more  complex.  As  a  scenario  becomes  increasingly  complex,  the  dynamics  between  people, 
tools  and  tasks  become  exponentially  greater,  outstripping  the  capabilities  of  a  computational 
modelling  application.  Therefore,  there  is  a  trade-off  between  the  complexity  of  a  scenario  and 
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the  ability  of  a  computational  model  to  support  it.  The  abilities  and  limitations  of  computational 
models  are  discussed  in  Section  5.2.6  below. 

The  scenarios  developed  for  this  project  have  the  potential  to  be  very  complex,  constructed  as 
they  are  of  up  to  three  teams  of  3  people  each.  It  is  unclear  whether  other  applications  have 
attempted  to  model  such  scenarios,  and  it  was  established  that  limited  previous  work  has 
modelled  teams-of-teams.  Thus,  it  is  likely  to  require  a  significant  effort  to  model  the  scenarios 
described  in  this  report  to  the  level  of  fidelity  required  for  research  purposes. 

5.1.5  Further  Development 

In  their  present  state,  the  developed  scenarios  provide  a  general  conception  of  how  events  would 
flow.  A  start  point  is  expected  to  trigger  the  situation  and  the  achievement  of  certain  states  will 
signify  the  end  point  (Figure  2).  In  between  the  start  and  end  points,  a  series  of  modifier  and 
distracter  inputs  will  be  introduced  to  simulate  real  life  events  and  to  provoke  various  team 
processes  and  (Figure  3).  The  next  step  in  developing  the  scenarios  is  to  therefore  specify 
potential  inputs,  expected  team  activities  and  expected  outputs.  Table  10  below  identifies 
questions  that  should  be  answered  regarding  possible  inputs,  expected  activities  and  expected 
outputs  to  provide  further  detail  to  the  scenarios. 


Table  10:  Further  Development  of  Scenarios 


Input 

Team  Response/Activity 

Output 

Modifier 

Input 

Time  of  the  Input 

What  is  the  input/its 
content? 

Input  recipient(s) 

Input  Medium 

Expected  response/ 
activity 

Additional  inputs  required? 

What  tools  will  be  used? 

Is  feedback  delivered  to 
the  originator  of  the  input? 

What  MOPs  will  be 
collected? 

Expected  time  of  output 

Expected  content  of  output 

Recipient(s)  of  output 

Output  medium 

What  MOPs  will  be 
collected? 

1 

2 

3... 

...n 

To  specify  the  details  of  the  inputs,  the  following  questions  can  be  asked:  What  is  the  content  of 
the  input;  who  is  the  input  intended  for,  and  in  what  medium  is  the  input  introduced?  The 
introduction  of  the  input  will  then  trigger  teams  to  work  together  on  a  task.  Once  the  inputs  have 
been  identified  in  detail  (this  process  can  be  facilitated  by  populating  Table  10),  the  resulting 
expected  output(s)  can  be  assumed.  Once  experimenters  know  what  types  of  outputs  are  likely  to 
occur,  s/he  can  choose  the  suitable  Measures  of  Performance  (MOPs)  and/or  Measures  of 
Effectiveness  (MOEs).  For  example,  if  the  input  is  a  telephone  call  to  notify  the  Commander  of 
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CanadaCOM  of  the  availability  of  new  resourees,  then  the  expected  output  is  mobilization  of 
these  resources  in  order  to  accomplish  the  mission.  An  MOP  could  be  the  amount  of  time  that 
passed  between  receipt  of  the  input  to  when  the  resources  were  employed.  By  outlining  an  input 
and  the  expected  team  activities  and  expected  outputs,  MOPs  and  MOEs  can  be  established.  The 
benefit  of  formulating  inputs,  MOPs  and  MOEs  ahead  of  time  is  that  the  experimenter  is  aware 
of  the  processes  that  should  take  place,  and  can  therefore  take  a  systematic  approach  to  data 
collection  and  research.  When  performance  differs  between  two  teams  carrying  out  the  same 
scenario  under  different  conditions,  the  experimenter,  having  adequately  defined  the  scenario  in 
the  beginning,  will  be  certain  that  the  observed  effect  is  due  to  their  experimental  manipulation, 
rather  than  random  differences. 

5.1.6  Limitations  of  the  developed  scenarios 

The  scope  of  this  contract  did  not  allow  the  HSI®  team  to  develop  fully  formed  team 
experimental  scenarios  that  could  effectively  function  as  they  presently  stand.  The  purpose  of 
this  contract  was  to  present  options  for  team  experimental  scenarios  that  could  then  be  further 
explored. 

Another  reason  that  the  scenarios  were  not  developed  in  detail  was  a  lack  of  publicly  available 
information.  CanadaCom  and  JTE  are  relatively  new  CE  organizations  and  therefore  it  was 
difficult  to  draw  conclusions  about  who  is  involved  in  each  team  and  how  the  teams  would  work 
together.  Similarly  it  was  unclear  as  to  what  the  relationship  between  DND  and  an  OGD  would 
be  since  new  organizations,  such  as  CanadaCOM  and  PSEPC  have  not  had  the  opportunity  to 
work  together  in  the  emergency  situations  depicted  by  the  scenarios  (in  fact,  they  have  only  had 
limited  opportunity  to  work  together  at  all).  Euture  work  with  SMEs  is  therefore  recommended 
to  help  bridge  these  gaps  in  knowledge. 

5.2  Comparison  of  Computational  Modelling  Applications 

As  can  be  seen  from  Table  9  and  Annex  E,  a  number  of  computational  models  scored  on  greater 
than  10  out  of  14  criteria  (IPME,  IMPRINT,  MIDAS,  D-OMAR,  SOAR,  BRAHMS,  CAST, 
STEAM,  DDD,  TOD,  C3TRACE,  ACHIMEDES,  RESA  and  JIMM).  However,  not  all  of 
these  received  ‘yes’  answers  to  the  critical  criteria  of  modelling  the  specified  team  tasks,  the 
specified  team  interaction,  HSI  interventions,  team  strategies  and  team  performance.  CAST  and 
ACHIMEDES  did  not  satisfy  all  these  criteria  while  still  scoring  above  10/14,  while  GLEAN, 
which  did  not  score  greater  than  10/14,  did  support  these  criteria.  The  answer  to  the  question  of 
which  one  is  the  best  is  dependent  upon  a  lot  of  variables.  What  is  it  that  we  are  trying  to  predict 
about  team  processes?  What  precise  questions  are  we  trying  to  answer?  Also,  how  complex  is 
the  environment  we  are  trying  to  model?  Is  the  team  work  co-located  or  remote?  How  big  is  the 
team?  How  much  data  do  we  have?  How  much  time  and  money  do  we  have?  One  is  not  really 
better  than  the  other  -  they  are  simply  different. 

Most  computational  models  were  designed  with  the  aim  to  model  basic  cognitive  abilities  or 
activities.  The  higher  level  cognitive  abilities  (e.g.  working  memory,  decision  making)  on  which 
team  processes  are  predicated  are  subjects  of  a  world-wide  on-going  research.  At  present,  as  one 
questionnaire  respondent  has  indicated,  none  of  the  existing  computational  models  of  human 
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behaviour  could  adequately  be  used  for  purposes  outlined  by  this  project.  Bearing  in  mind  the 
different  types  of  computational  models  described  above,  it  is  also  notable,  however,  that  IPME 
falls  into  the  category  of  computational  task  network  applications  that  result  in  information  about 
cognitive  measures  (e.g.  workload,  error,  etc.),  but  do  not  set  out  to  faithfully  model  cognition. 
Because  this  class  of  application  models  tends  to  generate  insights  into  cognition,  it  is  likely  more 
amenable  to  modelling  the  scenarios  being  generated  in  this  project. 

Most  tools  have  the  capability  of  being  linked  together  with  some  effort  for  specific  applications. 
For  example,  IMPRINT  has  been  linked  up  with  IPME/MicroSaint;  D-OMAR  together  with 
some  military  tools;  Apex  has  been  linked  into  MIDAS,  etc.  There  is  also  a  need  to  integrate 
models  with  different  strengths  to  complement  with  each  other,  for  example,  integration  of 
IMPRINT  and  ACT-R  to  combine  strengths  of  task  network  and  cognitive  modelling.  C3TRACE 
is  another  example  of  this  type  of  integration.  C3TRACE  takes  the  basic  task  network  modelling 
approach  and  adds  an  information-weighted  decision  making  algorithm.  Integration  of  CAST  and 
DDD  is  an  example  of  integrating  domain  independent  multi-agent  architecture  with  military  C2 
simulation  software.  The  CAST  architecture  was  extended  to  replace  some  or  all  of  the  players  in 
a  DDD  simulation  task. 

The  stability  of  a  modelling  tool  depends  on  the  match  between  the  tool  and  the  application 
domain.  No  tool  is  100%  stable  but  its  stability  is  mainly  dependant  on  the  severity  of  the 
adaptations  and/or  extensions  that  are  needed  for  applying  the  tool  to  the  desired  domain.  Most 
tools  are  applicable  outside  of  their  originally  designed-for  applications.  For  example,  using 
MIDAS  for  helicopter  applications  is  solid.  For  experiments  in  outer  space,  MIDAS  may  be  a 
little  less  solid  but  with  some  work,  it  can  be  applied  successfully. 

Given  the  criteria  used  to  assess  each  model,  it  is  felt  that  IPME  represents  the  best  value  for 
money  because  it  does  everything  that  is  required  of  it  (by  this  project)  and  is  in  a  continual 
process  of  improvement  in  whatever  manner  DRDC  sponsors.  It  is  also  one  of  the  classes  that 
have  been  developed  explicitly  to  model  tasks,  rather  than  to  faithfully  model  cognition.  And 
while  there  are  IPME  components  intended  to  mimic  cognition  in  terms  of  output,  they  are 
representative  in  terms  of  the  outputs  and  not  in  terms  of  the  “inner  workings”  of  cognition  (e.g. 
that  cognition  occurs  via  a  multiple  channel,  limited  capacity,  information  processor).  This 
makes  it  ideal  for  future  team  research  purposes.  Further,  there  is  a  large  defence  research  and 
industrial  community  who  are  comfortable  in  using  IPME,  and  any  attempt  to  introduce  a  new 
modelling  application  would  likely  result  in  significant  time  spent  familiarising  with  the  new 
application  before  insightful  outputs  were  produced.  Finally,  it  is  felt  that  the  windows-based 
interface  of  IPME  lends  itself  to  rapid  model  development,  more-so  than  the  others.  The 
combination  of  windows  dialogues  with  code,  renders  IPME  even  more  flexible  and  compatible 
with  the  needs  of  team  research. 

5.2.1  Implications  for  the  Canadian  Forces 

As  will  be  apparent  elsewhere  in  this  report,  the  development  of  a  strong  computational  model  of 
team  performance  will  provide  value  to  the  CF  in  a  number  of  areas.  First  and  foremost,  it  has 
been  demonstrated  in  other  projects  for  the  CF  that  having  a  good  computational  model  allows  an 
interested  party  to  assess  the  likely  impact  of  human-systems  integration  interventions.  For 
instance,  Matthews  et  al  (2005)  developed  an  IPME  model  to  assess  the  likely  impact  of 
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automating  the  sanitisation  task  of  sonar  operators.  The  automation  of  this  task  led  to  much  more 
time  spent  examining  the  beams  of  the  towed  array  for  real  targets  and  less  time  spent 
considering  clutter,  which  would  lead  to  greater  operational  effectiveness  in  the  CF.  Further, 
because  of  the  task  network  simulation,  this  investigation  was  run  thousands  of  times,  rather  than 
just  a  few,  and  cost  a  trivial  amount  when  compared  to  actually  building  the  system  and  installing 
it  in  a  simulator  or  a  real  ship. 

A  computational  model  is  also  useful  for  identifying  where  bottlenecks  could  occur  in  the  team. 
By  modelling  an  existing  structure  and  roles  the  model  will  show  where  the  highest  workloads  or 
greatest  time  constraints  occur  and  will  indicate  what  tasks  are  delayed  and/or  shed,  and  thus 
what  the  knock-on  impact  on  other  team  members  might  be.  This  allows  a  more  focused 
approach  to  developing  support  systems  or  making  other  HSI  interventions  to  do  with  training, 
team  or  organisational  structures. 

In  supporting  development  of  new  systems,  a  good  computational  model  will  also  generate 
detailed  requirements  for  system  performance  that  supports  operator  performance.  In  many 
systems  the  operator  will  wait  for  system  responses.  Delayed  responses  adversely  impact  the 
operator’s  performance.  Further,  the  operator  will  not  necessarily  realise  the  system  is  working 
and  the  operator  will  attempt  to  hurry  the  system  along.  This  can  result  in  the  system  hanging, 
or  additional  inputs  being  made  extremely  rapidly  once  the  system  is  freed  up,  leading  to  errors. 
Setting  appropriate  system  performance  requirements  that  support  the  operator  can  be  derived 
and  tested  from  computational  models. 

A  final  benefit  to  the  CF  is  a  more  effective  use  of  their  personnel.  Currently,  there  is  great 
demand  on  the  CF  to  provide  suitably  qualified  personnel  to  participate  in  experiments  or  test 
new  systems,  equipment,  procedures,  etc.  With  a  suitably  developed  computation  model,  the 
demand  will  continue,  but  it  would  only  be  in  support  of  systems  or  developments  that  have 
shown  significant  benefit  in  the  computational  model.  This  would  reduce  the  ongoing  demand 
for  personnel.  However,  if  intact  teams  could  not  be  obtained  for  such  testing,  it  may  also  be 
possible  to  use  the  computational  model  as  an  additional  team  member.  This  approach  has  been 
investigated  by  DRDC  Toronto  in  the  context  of  helicopter  deck  landing  aboard  the  Halifax-Class 
frigate  (Lamoureux  et  al.,  2004),  and  in  the  provision  of  computer-generated  armoured  fighting 
vehicles  (Mekdeci,  2004).  Thus,  CF  personnel  could  be  used  much  more  effectively  to  support 
projects. 

5.2.2  Implications  for  Scenarios 

In  the  SOW  for  this  project,  the  SA  posed  a  number  of  specific  questions  referring  to  the  manner 
in  which  the  scenario  developed  above  (Section  3.2.3)  could  be  implemented  in  a  computational 
modelling  application.  Each  of  these  questions  is  answered  individually  below. 

Propose  how  the  scenario  developed  in  Section  3.2.3  above  may  be  implemented  as  an  IPME  task 
network  model. 

The  scenario  would  be  subject  of  a  task  analysis.  This  task  analysis  would  describe  the  structure 
and  interdependence  of  all  tasks,  uncover  the  various  inputs  and  outputs  of  each  task  (including 
who  the  actor  or  recipient  would  most  likely  be,  with  a  nominated  alternative),  along  with  the 
workload  and  expected  task  frequencies  and  completion  times,  and  other  task  parameters  such  as 
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error  rates  and  eonsequenees.  The  seenario  could  then  be  implemented  in  IPME  as  a  task 
network  model.  Each  task  in  the  network  would  then  be  subject  to  the  standard  IPME 
implementation  of  task  shedding,  delaying,  etc.  subject  to  the  workload  being  experienced  by  the 
actor.  The  task  network  model  would  be  combined  with  the  scenario  event  monitor  which  would 
create  time-  and  event-based  cues  that  would  exercise  the  various  elements  of  the  network. 

Propose  how  the  scenario  developed  above  may  be  implemented  in  an  IPME  crew  model. 

The  crew  model  is  a  list  of  operators  (or  humans)  who  carry  out  tasks  in  the  task  model.  The 
crew  model  always  exists.  An  IPME  model  must  always  have  at  least  one  crew  member.  Crew 
members  can  be  modeled  in  whatever  degree  of  detail  desired.  The  crew  model  includes  three 
basic  groups:  properties  (hands/feet/fingers,  etc.),  traits  (height,  weight,  cognitive  ability,  etc.), 
and  states  (time  since  slept,  temperature,  etc.).  Instead  of  using  the  task  network  approach  of 
assigning  individuals  to  a  task,  the  crew  model  enables  IPME  to  simultaneously  monitor  and 
assign  workload  for  multiple  individuals  within  a  common  scenario,  while  also  accounting  for 
interaction  effects  between  properties,  traits  and  states.  Workload  can  be  assigned  dynamically 
to  different  operators  within  any  active  task  using  either  syntax  or  rigid  “rules”  (IPME  can  be 
programmed  or  the  built-in  windows  interface  can  be  used)  assigned  within  the  task  dialog.  In 
this  manner,  procedural  rules  or  skill/experience  restrictions  can  be  implemented  and  followed, 
but  the  tasks  can  be  carried  out  flexibly  in  the  same  manner  that  they  would  likely  be  carried  out 
by  real  teams. 

Propose  how  the  scenario  developed  above  may  be  implemented  in  an  IPME  environment  model, 
if  applicable. 

A  team  task  would  not  be  modeled  in  the  IPME  environment  model.  Rather,  the  environment 
model  is  something  that  can  interact  with  the  task  network  or  team  model  to  affect  performance 
(generally  time  to  complete  a  task  and  error  rates).  The  environment  model  allows  the  developer 
to  control  the  impact  of  external  events  on  the  scenario.  Eor  example,  light  conditions  can  be 
changed,  temperature  can  be  changed  (all  within  the  environment  model)  and  as  a  result  these 
values  can  be  stored  in  variables  used  to  influence  task  performance.  Eor  example,  task 
completion  times  can  be  calculated  according  to  daylight  (e.g.,  in  low  light  conditions,  task 
completion  times  double  or  error  probabilities  increase  by  a  factor  of  2).  If  a  team  task  takes 
place  at  night,  or  even  over  a  long  period  of  time  during  which  conditions  vary,  the  environment 
model  can  modify  task  success  accordingly.  A  team  task  may  also  suffer  extremes  of  cold  or 
precipitation  that  may  positively  or  negatively  affect  performance. 

Eurther  to  these  specific  questions,  it  is  questionable  whether  the  computational  model  has  any 
direct  implications  for  the  scenario  being  developed.  In  so  far  as  the  computational  model  should 
be  technically  able  to  represent  the  scenario  being  developed,  the  scenario  will  need  to  be  of 
sufficient  simplicity  to  facilitate  this  representation,  but  this  is  likely  to  be  of  secondary  priority 
to  the  creation  of  scenario  that  will  support  the  team  research  aims  of  DRDC  Toronto.  IPME 
should  not,  however,  impose  these  constraints  on  the  scenario  because  it  is  a  powerful  modelling 
application  with  a  highly  flexible  syntax  option.  The  limitation  is  likely  to  be  speed,  processing 
power  and  memory  in  the  host  machine,  which  itself  is  a  limitation  that  various  stakeholders  are 
working  to  overcome. 
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5.2.3  Implications  for  Team  Research 

In  the  SOW  for  this  project,  the  SA  posed  a  number  of  specific  questions  about  the  implications 
of  a  computational  model  for  team  research.  Each  of  these  questions  is  answered  individually 
below. 

Identify  key  similarities  and  differences  between  the  proposed  implementation  and  previous 
implementations  of  team  models  in  IPME. 

Previous  implementations  of  teams  in  IPME  do  not  consider  team  performance  as  a  variable  in 
task  performance.  Eor  instance,  previous  IPME  models  have  assumed  perfect  team  performance 
(as  opposed  to  perfect  individual  performance  as  measured  by  shed,  interrupted  and  delayed 
tasks).  However,  ‘real’  team  members  are  not  always  available  to  assist  each  other  or  receive 
information  from  each  other.  The  proposed  implementation  would  be  built  using  a  theoretical 
model  of  team  performance  that  includes  variables  to  account  for  real  variations  in  team 
performance.  In  this  manner,  the  output  of  the  model  would  more  closely  approximate  actual 
performance  and  could  thus  be  used  for  predictive  purposes. 

Propose  the  use  of  an  existing  or  a  new  model  of  workload  in  IPME  for  modelling  the  scenario 
developed  above,  and  provide  the  rationale  for  choosing  this  workload  model. 

The  IPME  model  could  generate  significant  insights  into  the  cognitive  impacts  of  the  scenario  on 
operators.  With  IPME,  this  insight  has  often  been  related  to  workload  (at  least,  at  a  summary 
level).  The  VACP  model  is  an  appropriate  workload  model  in  IPME  for  evaluating  team 
performance.  VACP  can  be  used  in  conjunction  with  the  crew  model  to  evaluate  instantaneous 
workload  for  individual  team  members,  and  aggregate  workload  for  the  team  as  a  whole  (or 
entity).  VACP  is  considered  the  ‘basic’  IPME  workload  model  though.  To  better  approximate 
‘real’  team  performance,  the  POP/IP  model  of  workload  is  proposed.  POP/IP  (Prediction  of 
Operator  Performance/Information  Processing)  integrates  Qinetiq’s  (POP)  model  with  DRDC’s 
IP/PCT  (Perceptual  Control  Theory)  models.  In  particular,  the  POP/IP  model  includes  the 
concepts  of  structural  interference,  task  deferment,  task  shedding,  and  prospective  memory  from 
the  IP/PCT  model,  and  the  general  interference  from  the  POP  model.  The  model  is  intended  to 
reduce  the  potentially  substantial  effects  of  congestion  within  the  POP  model.  Workload  is 
calculated  from  the  task  scheduler,  interacting  with  the  workload  parameters  for  a  task.  A 
comparison  of  POP/IP  with  POP  and  IP/PCT  is  presented  in  Table  1 1  below. 


Table  1 1 :  Comparison  of  task  schedulers  for  different  IPME  workload  algorithms 


POP 

IP/PCT 

POP/IP 

Task  shed 

No 

Yes 

Yes 

Task  delayed 

Yes 

Yes 

Yes 

Task  Interrupted 

Yes 

Yes 

Yes 

Time  penalty 
(concurrent  task 
processing) 

Task  Demand  Multiplier 
(TDM) 

Time  penalty 

TDM 

Workload 

Operatorl  .workload 

Operatorl  .MeanTImePressure 

Operatorl  .workload 

Operatorl  .MeanTImePressure 
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POP 

IP/PCT 

POP/IP 

Structural  Interference 

Manual  output 

Visual,  auditory,  psychomotor 

Visual,  auditory,  psychomotor 

Task  Demand 

Input,  central,  output 

Visual,  auditory,  cognitive, 
psychomotor 

Input,  central,  output,  visual, 
auditory,  psychomotor 

Task  Priority 

Internally/externally  paced 

Multiplier 

Internally/externally  paced  and 
multiplier 

Additional  time  penalty 

Performance  shaping 
factor  (PSF) 

Time  performance  modifier 

PSF 

Forgetting 

No 

Yes 

Yes 

Short-term  memory 

No 

Yes 

Yes 

Interference 

coefficients 

No 

User  can  modify 

Fixed  values 

Compatible  task  pairs 

No 

Yes 

Yes 

As  apparent  in  the  table  above,  POP/IP  attempts  to  use  the  most  suceessful  and  desirable  features 
of  the  different  workload  algorithms. 


Returning  to  the  point  made  above  that  a  computational  model  allows  the  focused  development  of 
needed  tools,  the  computational  model  could  indeed  lead  to  the  focused  running  of  expensive 
‘man-in-the-loop’  team  experiments.  Although  founded  in  theory,  many  other  team  experiments 
have  really  proceeded  in  the  hope  that  something  insightful  would  result.  Any  hypothesis  being 
tested  has  been  necessarily  high-level  in  its  description.  By  developing  a  team  scenario  in  a 
computational  model,  experimental  manipulations  (i.e.  the  independent  variables)  can  be  made  to 
determine  what  manipulation  results  in  the  biggest  insights.  This  can  then  be  repeated  in  a 
‘human-in-the-loop’  trial  for  both  validation  of  the  computational  model  and  for  additional 
insights. 

The  common  assumption  when  considering  live  human  experiments  and  the  computational  model 
is  that  the  computational  modeling  is  conducted  first.  In  this  way,  it  can  inform  the  research. 
This  means  that  the  computational  model  must  be  built  and  exercised  a  significant  time  before  the 
human-in-the-loop  trial  is  scheduled,  in  order  that  its  outputs  can  be  considered  and  fed 
seamlessly  into  the  live  trial.  Except  for  validation  purposes,  it  is  unlikely  that  the  computational 
model  should  ever  be  created  after  the  live  trial  has  been  run. 

5.2.4  Implications  for  Human-Systems  Integration 

As  noted  above,  the  computational  models  can  provide  significant  benefit  for  HSI  purposes.  In 
particular,  the  use  of  computational  models  will  increase  the  confidence  that  a  proposed 
intervention  will  in  fact  provide  a  significant  beneficial  effect.  Further,  it  may  reduce  the 
demand  upon  the  CF  to  provide  personnel  to  test  new  HSI  interventions. 

Another  possible,  but  less  obvious,  benefit  to  human-systems  integration  is  the  opportunity  to 
ensure  scenarios  are  best  suited  for  testing  the  target  intervention.  Ideally,  every  system  in  the 
military  has  been  subject  to  a  Mission,  Function  and  Task  Analysis  (MFTA)  or  a  similar  sort  of 
analysis.  The  MFTA  leads  to  a  set  of  critical  mission  requirements  which  can  be  incorporated 
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into  a  scenario  to  serve  as  the  basis  for  live  testing.  With  revolutionary  designs  or  new 
capabilities,  the  critical  mission  requirements  may  not  fully  address  the  capabilities  of  the  new 
system.  A  computational  model  can  be  used  to  ereate  and  test  the  seenario  and  the  new  system 
before  the  CF  expends  significant  resources  staging  a  live  trial.  Such  a  test  would  involve 
baselining  the  scenario  before  running  the  scenario  with  the  new  system.  Depending  upon  the 
stated  capabilities  of  the  new  system,  the  analyst  will  determine  whether  the  model  sufficiently 
exereises  those  eapabilities.  If  it  does  not,  then  the  scenario  will  need  to  be  revised  and  tested 
again  before  it  is  incorporated  in  the  live  scenario.  This  approach  would  significantly  improve 
the  chances  of  an  insightful  trial  and  render  such  acceptance  testing  more  than  a  ‘rubber- 
stamping’  process. 

5.2.5  Further  Development 

In  the  SOW  for  this  project,  the  SA  posed  a  number  of  specific  questions  referring  to  the  manner 
in  which  IPME  could  be  improved.  Each  of  these  questions  is  answered  individually  below. 

Propose  changes  or  additions  to  IPME  that  will  increase  IPME’s  utility  for  modelling  the 
scenario  developed  above. 

In  general,  IPME  is  flexible  enough  to  accommodate  any  team  researeh  scenario  that  needs  to  be 
built,  provided  enough  time  and  data  is  available  to  create  the  model.  However,  implicit  in  this 
statement  is  an  aeknowledgement  that  an  IPME  model  takes  time  to  build  and  further  time  to 
calibrate  to  generate  valid  performance  outputs.  IPME  also  provides  the  raw  data  relating  to  task 
and  actor  performance  to  make  any  necessary  calculation  to  generate  predictive  insights  into  team 
performance.  Using  this  raw  data  IPME  could  output  an  overall  metric  of  team  performance, 
perhaps  generating  output  for  a  team  as  an  entity.  IPME  could  also  generate  measures  of 
situation  awareness  and  error  ‘critieality’  (i.e.  what  percentage  of  errors  are  considered  eritical  to 
mission  success).  Thus,  there  are  no  changes  or  additions  to  IPME  that  cannot  be  implemented 
in  the  eourse  of  a  suitably  scoped  project. 

Propose  changes  or  additions  to  IPME  that  will  increase  IPME’s  usability  for  modelling  the 
scenario  developed  above. 

To  speed  up  the  development  of  IPME  models,  it  is  desirable  to  implement  a  ‘library’  of  ‘typical’ 
teams  and  tasks.  Then  the  user  could  select  the  basic  team  or  task  strueture  and  modify  it  to 
his/her  own  requirements.  Such  a  hypothetical  team  model  could  be  a  component  of  the  crew 
model,  making  a  team  a  team  (in  the  sense  defined  by  Sartori  et  al.,  2006),  rather  than  a  group  of 
individuals.  Teams  could  vary  in  size  (e.g.  3  person,  5  person,  etc.)  and  degree  to  which  they 
are  expected  to  work  in  a  distributed  rather  than  a  co-loeated  fashion.  The  degree  to  whieh  they 
are  stable  teams,  as  opposed  to  ad-hoc  teams,  could  also  be  varied.  Likewise,  tasks  could  be 
created  in  a  generic  fashion  to  facilitate  the  rapid  development  of  scenario  speeific  task  networks. 
Tasks  could  comprise  various  combinations  of  input,  decision  making,  iteration,  feedback, 
output,  monitoring  and  time  pressure.  To  this  end,  Humansy stems  have  developed  a  generic 
model  of  process  control,  modeled  on  the  human  information  processing  model  of  Wickens 
(1984).  This  accommodates,  at  a  ‘micro’  level,  all  manner  of  cognitive  work  eonducted  by 
humans.  This  model  can  be  re-used,  within  each  task  if  necessary,  as  the  basis  of  a  task 
network.  One  further  improvement  refers  to  the  current  manner  in  which  models  interaet.  The 
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user  has  to  develop  a  model  in  IPME  as  a  separate  project  and  link  the  model  using  the  HLA 
protocol  so  the  models  on  different  computers  can  ‘speak’  to  each  other.  It  would  be  more 
convenient  to  add  additional  model  components  to  IPME  as  simple  plug-ins. 

Other  ongoing  improvements  to  IPME  sponsored  by  DRDC  and  other  compatible  applications 
(e.g.  TaskArchitect)  will  also  increase  the  utility  and  usability  of  IPME  and  render  it  more 
generally  complimentary  to  team  research. 

5.2.6  Limitations 

As  with  any  computational  modelling  application,  a  major  limitation  is  the  availability  of  suitably 
qualified  and  experienced  resources  to  do  the  modelling.  It  is  felt  that  the  Canadian  modelling 
community  is  very  strong  so  this  is  unlikely  to  be  a  significant  hurdle. 

With  respect  to  the  comparative  review  of  computational  models,  there  were  a  few  limitations  to 
the  conclusions  that  could  be  made.  Although  every  effort  was  made  to  qualify  each  yes/no 
judgment  with  a  description  as  to  the  quality  and  the  extent  to  which  the  model  actually  models  a 
particular  function,  the  reader  may  find  that  many  cells  in  Table  9  are  filled  simply  with  yes/no 
answers;  these  cells  are  either  self-explanatory  or  there  was  limited  available  relevant 
information.  In  those  cases,  the  reader  should  regard  the  entries  as  suggestive  and  consult  a  wider 
selection  of  references  for  a  more  detailed  description  of  model  capabilities.  The  only  way  that 
this  could  have  been  overcome  is  to  conduct  an  assessment  that  involved  modelling  the  same 
scenarios  in  different  applications.  This  would  obviously  have  been  prohibitively  expensive  in 
terms  of  time  and  effort. 

To  earn  a  “yes”  judgment,  either  the  documentation  and  other  literature  or  the  responses  from 
the  questionnaires  or  interviews  associated  with  the  model  had  to  indicate  or  describe  the  model’s 
capabilities  specifically  for  that  particular  function.  Sometimes  inferences  (educated  guesses) 
were  made  about  the  model’s  potential  capabilities  in  order  to  fill  in  the  cell.  Again,  this  could 
only  be  overcome  with  a  detailed  comparative  assessment. 

The  information  in  Annex  E  does  not  represent  a  “scorecard”  with  which  to  rate  the  merits  of 
computational  models,  but  a  composite  ‘score’  that  reflected  the  relative  weights  of  the  different 
criteria  would  have  been  helpful.  However,  this  would  have  had  no  validity  and  limited 
reliability  (analyst  perspectives  would  likely  have  been  wildly  variable)  so  no  attempt  was  made 
to  create  one. 

On  balance,  however,  it  is  felt  that  within  the  constraints  of  this  work,  there  were  few  limitations 
and  that  IPME  is  a  worthy  application  to  use  as  part  of  the  toolkit  for  team  research.  In 
summary,  the  current  state  of  the  art  in  computational  modelling  seems  to  indicate  that  IPME  is 
the  most  appropriate  tool  for  modelling  team  research  scenarios.  Eurther,  the  unique  and 
extensive  community  of  IPME  users  in  DRDC  and  Canadian  industry  at  large  makes  it  the 
logical  choice  for  such  efforts. 
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Conclusions 


This  work  investigated  the  team  research  literature  for  existing  experimental  scenarios,  and  the 
computational  modelling  literature  for  modelling  platforms.  Based  on  the  review,  no  pre-existing 
team  research  scenarios  were  appropriate  for  the  purposes  outlined  by  DRDC.  With  respect  to 
computational  modelling,  it  was  concluded  that  the  employment  of  IPME  does  indeed  meet  the 
requirements  of  DRDC  and,  further,  there  are  minimal  improvements  that  can  be  made  to  that 
platform,  beyond  enhancing  the  manner  in  which  models  can  be  networked  in  order  to 
accommodate  large,  complex  teams  performing  complex  tasks. 

The  scenario  development  work  resulted  in  three  experimental  scenarios:  a  natural  disaster;  a 
terrorist  threat;  and  an  influenza  pandemic.  These  scenarios  all  exercise  joint,  interagency  and 
multidisciplinary  aspects  of  teamwork,  and  exhibit  a  mixture  of  ad-hoc  and  fixed,  and  distributed 
and  co-located  teams.  The  scenarios  focus  on  operational  level  work,  rather  than  tactical  level 
work.  The  scenarios  were  developed  to  a  coarse  level  of  detail,  but  a  template  was  also 
developed  to  assist  in  further  developing  the  detail  for  each  scenario.  The  scenarios  exhibit  a 
medium  level  of  fidelity  that  while  offering  experimental  control.  The  scenarios  permit  team 
research  factors  uncovered  by  Sartori  et  al  (2006)  to  be  addressed,  either  through  experimental 
manipulation  or  through  measures  of  performance.  This  means  the  scenarios  are  inherently 
flexible  and  responsive  to  the  research  needs  of  DRDC  Toronto. 

It  is  recommended  that  three  follow-on  pieces  of  work  be  undertaken  as  soon  as  is  reasonably 
practicable: 

1.  Develop  the  detail  for  the  three  scenarios.  This  will  include  identifying  all  the  inputs  to 
the  scenario  and  also  the  expected  actions  and  their  outputs.  A  template  has  been 
provided  for  this  purpose. 

2.  Build  the  baseline  IPME  model  for  each  scenario.  This  will  have  to  wait  until  at  least 
one  scenario  is  fully  realised. 

3.  Draw  up  detailed  plans  for  the  development  of  an  experimental  laboratory  that  will 
accommodate  the  scenarios.  These  plans  should  include  details  of  how  participants  will 
interact  with  other  participants  and  the  system. 

Subsequent  to  this,  human-in-the-loop  experiments  should  be  conducted  in  the  experimental 
facility  to  provide  a  baseline  for  the  IPME  model  and  to  perform  preliminary  model  validation, 
before  new  research  begins. 
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Annex  A:  Evaluation  of  Reviewed  Scenarios 


Uxxmmsystems^ 
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No.1 

Scenario  name: 

Search-and-Rescue  Mission  in  Antarctica 

Platform  name  (asterisk 

‘NASA  Ames  Centre  -  Distributed  Research  Facilities  (based  on  Optima  Technology's 

if  reviewed  in  platform 

simulation  software) 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

17 

Operational 

Relevant 

l~ 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

w 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

17 

r 

r 

Medium 

1“ 

1“ 

1“ 

Task  mental  model 

R 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

17 

r 

r 

Explicit 

r 

r 

r 

1 .2  Team  History 

Centralized  network 

17 

1“ 

n 

Ad  Hoc 

p 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

k 

r 

r 

Implicit 

r 

r 

r 

Co-located 

r 

r 

r 

Explicit 

r 

W 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

W 

r 

r 

Uncertainty 

w 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

17 

r 

Resource  Allocation 

r 

17 

r 

Cognitive 

r 

17 

r 

Time  pressure 

W 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

r 

r 

Automation 

17 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

1^ 

r 

r 

Observers 

17 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

17 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments: 

N/A 

Reference(s): 

http://wvwv.nasa.gov/centers/ames/research/technology-onepagers/distributed-team- 

decision.html 

http://vwvw.nsbri.org/Research/Projects/viewsummary.epl?pid= 

170 

http://gtresearchnews.gatech.edu/reshor/rh-ss03/sp-team.html 
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No.2 

Scenario  name:  Team  Resource  Allocation  Problem  in  Emergency  Criss  Management 


Platform  name  (asterisk 

'NeoCITIES 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

Strategic 

r 

Highly  relevant 

13 

Operational 

K7 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

17 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

13 

r 

1“ 

Team  mental  model 

r 

r 

Medium 

r 

1“ 

1“ 

Task  mental  model 

I7 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

17 

r 

r 

Implicit 

1“ 

r 

r 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

1“ 

r 

r 

Ad  Hoc 

17 

r 

r 

Decentralized  network 

17 

r 

r 

Fixed 

r 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

17 

r 

r 

Implicit 

13 

r 

r 

Co-located 

r 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

17 

r 

r 

Uncertainty 

17 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

1“ 

Resource  Allocation 

r 

17 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

13 

r 

1“ 

Automation 

13 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

1“ 

1“ 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments:  At  its  core,  NeoCITIES  is  a  team  resource  allocation  problem  designed  to  mimic  the 

emergent  situations  that  comprise  real-life  emergencies  and  measure  decision-related 
outputs  in  a  virtual  environment.  The  simulation  emulates  the  complex  functions 
involved  in  the  resource  management  of  a  city’s  emergency  services.  Crisis 
management  is  conducted  through  the  joint  interaction  of  three  distinct  teams:  a  Police 
team,  a  Fire/EMS  team,  and  a  Hazardous  Materials  team,  each  consisting  of  an 
information  manager  (IM)  and  resource  manager  (RM). 

Reference(s): 

http://minds.ist.  psu.edu/index.php?option=com_content&task=view&id=54&ltemid=67 

R.E.T.  Jones,  M.D.  McNeese,  E.S.  Connors,  T.  Jefferson,  Jr.,  D.L.  Hall,  (  2004).  "A 
distributed  cognition  simulation  involving  homeland  security  and  defense:  The 
development  of  NeoCITIES,”  Proceedings  of  the  Human  Factors  and  Ergonomics 
Society  48th  annual  meeting  (New  Orleans,  Louisiana),  Santa  Monica,  CA:  Human 
Factors  and  Ergonomics  Society,  pp.  631-634. 

McNeese,  M.D.  et  al  (2005).  The  NeoCITIES  Simulation:  Understanding  the  design 
and  experimental  methodology  used  to  develop  a  team  emergency  management 
simulation.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society  49th  Annual 
meeting,  pp591-.594 
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No.3 

Scenario  name:  Uninhabited  Air  Vehicle  (UAV)  Ground  Control  Operations  to  Taking  Reconnaissance 

Photos 


Platform  name  (asterisk  ’'Synthetic  Task  Environment  (STE)  in  Cognitive  Engineering 

Research  on  Team 

if  reviewed  in  platform  Tasks  (CERTT)  Lab 
report): 

Level  of  activity 

Relevance 

strategic  1 

Highly  relevant 

1“ 

Operational  ^ 

Relevant 

It/ 

Tactical  ^ 

Somewhat  relevant 

r 

Purpose 

Training  |~ 

Research 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

It? 

r 

r 

Team  mental  model 

1“ 

r 

17 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

17 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

1“ 

1“ 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

1“ 

r 

1“ 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

Fixed 

r 

r 

Hierarchical  network 

r 

r 

r 

1 .3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

r 

Co-located 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

r 

1“ 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

r 

r 

r 

Automation 

(;? 

Conjunctive 

It? 

r 

r 

Self-Report 

P 

Disjunctive 

r 

r 

r 

Cbservers 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

Sequential 

r 

r 

r 

Team  Performance 

[7 

Reciprocal 

r 

r 

r 

Comments:  The  CERTT  Lab  houses  a  Synthetic  Task  Environment  (STE)  for  studying  team 

cognition.  The  premier  task  in  this  STE  is  a  simulation  of  Uninhabited  Air  Vehicle 
(UAV)  Ground  Control  operations. In  the  UAV-STE  a  three-person  team  controls  the 
UAV  for  the  purpose  of  taking  reconnaissance  photos.  The  task  is  based  on  the  actual 
operations  of  the  Air  Force’s  Predator  UAV.  The  synthetic  task  environment  in  made 
up  of  four  industrial  Participant  Consoles  and  one  Experiment  Console. The  three  team 
members  have  a  distinct  role:  AVO  (Air  Vehicle  Operator),  PLO  (Payload  Operator) 
and  DEMPC  (Data  Exploitation,  Mission  Planning  and  Communications  Operator) 


Reference(s): 


http://www.certt.com/ 
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No.4 

Scenario  name:  Command  and  Controi  Simuiation 

unfriendiy  entity 

Platform  name  (asterisk  *DDD 
if  reviewed  in  piatform 
report): 

Level  of  activity 

strategic  1 

Operational  1^ 

Tactical  1 

Purpose 

Training 

Research  W 

-  Defending  a  region  against  invasion  from 

Relevance 

Highly  relevant  17 

Relevant  1 

Somewhat  relevant 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

1“ 

r 

Task  mental  model 

W 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

17 

r 

Explicit 

r 

W 

r 

1.2  Team  History 

Centralized  network 

1“ 

1“ 

1“ 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

w 

r 

r 

Fixed 

w 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

t7 

1“ 

r 

Co-located 

w 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

i~ 

1“ 

i~ 

2.1  Task  Complexity 

Feedback 

r 

w 

r 

Uncertainty 

w 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

1“ 

1“ 

1“ 

Resource  Allocation 

r 

r 

r 

Cognitive 

n? 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

i~ 

1“ 

1“ 

Automation 

17 

Conjunctive 

17 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

17 

1“ 

[“ 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

1“ 

Sequential 

r 

r 

r 

Team  Performance 

Reciprocal 

r 

r 

r 

Comments: 

Reference(s): 


N/A 

http://www.aptima.com/Projects/Distributed_Dynamic_Decision_making.html 


Articie:  Team  Learning:  Coiiectiveiy  Connecting  the  Dots,  Eiiis,  et  ai  (secondary  ref: 
Miiier,  Young,  Kieinman  &  Serfaty,  1998) 


http://www.dodccrp.org/events/2005/10th/CD/papers/358.pdf 
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-r 

Nol 

Scenario  name:  Combat  Information  Center  -  Monitoring  radar  display  for  targets 


Platform  name  (asterisk 

‘Tactical  Navy  Decision  Making  System  (TANDEM) 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

Strategic 

r 

Highly  relevant 

r 

Operational 

r 

Relevant 

a 

Tactical 

y 

Somewhat  relevant 

r 

Purpose 

Training 

y 

Research 

r 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

y 

r 

r 

Team  mental  model 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

y 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

17 

r 

r 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

y 

r 

r 

Fixed 

y 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physicai  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

r 

r 

Co-iocated 

y 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

y 

r 

r 

Uncertainty 

y 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

1“ 

17 

r 

Time  pressure 

r 

y 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

r 

17 

r 

Automation 

r 

Conjunctive 

r 

r 

r 

Self-Report 

y 

Disjunctive 

r 

r 

r 

Observers 

y 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

y 

Reciprocal 

r 

r 

r 

Comments:  TANDEM  was  designed  to  be  a  more  ecologically  valid  simulation  of  a  command, 

control,  and  communication  environment;  rather  than  use  synthetic  work  it  employs 
tasks  that  are  closer  to  the  real-  life  counterpart  of  a  combat  information  center. 
Decision-making  skills  require  information-sharing  among  one  to  three  participants,  as 
decisions  must  be  made  based  on  provided  information  regarding  unknown  contacts. 
For  each  contact,  characteristics  -  type,  threat  level,  and  intent  -  must  be  determined, 
and  fifteen  pieces  of  information  are  required  to  make  a  ruling  on  each  contact  (five 
pieces  for  each  characteristic).  Each  participant  receives  information  pieces,  some  of 
which  might  be  conflicting  or  ambiguous.  Task  characteristics  such  as 
interdependence,  time  pressure,  and  work  load  can  be  examined,  and  the  scenario  is 
reconfigurable.  However,  TANDEM  does  not  require  the  integration  of  new  or 
changing  information  overtime;  participants  are  equipped  with  the  same  knowledge 
set  for  the  duration  of  the  session. 

Reference(s):  Dwyer,  D.,  Hall,  J.,  Voipe,  C.,  Cannon-Bowers,  J.  A.,  &  Salas,  E.  (1992).  A 

performance  assessment  task  for  examining  tactical  decision-making  under  stress 
(Report  No.  92-002).  Orlando,  FL;  Naval  Training  Systems  Center. 

http;//www.  iwm-kmrc.de/workshops/sim2004/pdf_filesA/anBerlo.  pdf 

http;//usl. sis.  pitt.edu/ulab/pubs/HFES99LHLR. pdf 
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No.6 

Scenario  name:  Naval  Surveillance  and  Threat  Assessment  Operation 


Platform  name  (asterisk 

*Team  and  Individual  Tactical  Assessment  Network  (TITAN) 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

r 

Operational 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

17 

Purpose 

Training 

[7 

Research 

r 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

r 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

17 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

r 

r 

Co-located 

[7 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

15? 

r 

r 

Automation 

15? 

Conjunctive 

1“ 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

i;? 

r 

r 

Individual  Performance 

15? 

Sequential 

r 

r 

r 

Team  Performance 

[7 

Reciprocal 

r 

r 

r 

Comments:  N/A 

Reference(s):  The  effect  of  individual  differences  in  cognitive  styles  on  decision-making 

accuracy.Blais,  Ann-Renee;  Baranski,  J.V.  ;  Thompson,  M.M.  (DRDC-Toronto) 


http://www.toronto.drdc-rddc.gc.ca/publications/factsheets/f09_e.html 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-7 


HUMANS)Sr£  l/S 

-r, 


No.7 

Scenario  name: 

Tank  Battle  Game 

Platform  name  (asterisk 

*Bolo 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

r 

Operational 

r 

Relevant 

1“ 

Tactical 

1^ 

Somewhat  relevant 

Purpose 

Training 

r 

Research 

1^ 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r? 

1“ 

n 

Team  mental  model 

1“ 

1“ 

1“ 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

i~ 

1“ 

i~ 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

w 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

r? 

r 

Co-located 

w 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

w 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

w 

r 

Cognitive 

1“ 

1“ 

1“ 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

r 

r 

Automation 

r? 

Conjunctive 

1“ 

1“ 

1“ 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

[“ 

1“ 

1“ 

Individual  Performance 

17 

Sequential 

r 

r 

r 

Team  Performance 

P 

Reciprocal 

r 

r 

r 

Comments:  N/A 

Reference(s):  Article:  The  relationship  of  team  goals,  incentives,  and  efficacy  to  strategic  risk,  tactical 

implementation,  and  performance;  Don  Knight,  Cathy  C.  Durham,  Edwin  A.  Locke 


http://www.lgm.com/bolo/ 

http://www.twinforces.com/tf/bolo3d.htm 


A-8 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


HUMANS)Xr£.l/S 


No.8 


Scenario  name: 

Naval  Combat  Experience 

Piatform  name  (asterisk 

*Dangerous  Waters 

if  reviewed  in  platform 

report): 

Levei  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

r 

Operational 

17 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

W 

Purpose 

Training 

r 

Research 

w 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

17 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

1“ 

i~ 

1“ 

Explicit 

r 

r 

r 

1 .2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

17 

r 

r 

Hierarchical  network 

r 

r 

r 

1 .3  Physical  Distribution 

3.3  Coordination 

Distributed 

17 

r 

r 

Implicit 

r 

r 

r 

Co-located 

r 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

17 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

17 

r 

r 

Automation 

17 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

17 

Sequential 

r 

r 

r 

Team  Performance 

r 

Reciprocal 

r 

r 

r 

Comments:  N/A 

Reference(s):  Dangerous  Waters  Manual 


http://www.strategyfirst.com/en/games/DangerousWaters/ 

http://www.scs-dangerouswaters.com/ 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-9 


No.9 

Scenario  name:  Helicopter  flight  simulator 

Platform  name  (asterisk  *Longbow2 


if  reviewed  in  platform 
report): 

Level  of  activity 

strategic 

Operational 

Tactical 

Purpose 

Training 

Research 

1“ 

K? 

I? 

r 

[7 

Relevance 

Highly  relevant 

Relevant 

Somewhat  relevant 

1“ 

15 

r 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

1“ 

1“ 

Team  mental  model 

1“ 

15 

1“ 

Medium 

1“ 

1“ 

1“ 

Task  mental  model 

r 

17 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

1” 

15 

n 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

1” 

1“ 

n 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

17 

r 

Fixed 

17 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

1” 

15 

1” 

Co-located 

17 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

r5 

1“ 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

17 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

1“ 

1“ 

1“ 

Resource  Allocation 

r 

r 

r 

Cognitive 

1“ 

1“ 

1“ 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

1“ 

1“ 

Automation 

r5 

Conjunctive 

1“ 

1“ 

1“ 

Self-Report 

17 

Disjunctive 

r 

r 

r 

Observers 

17 

Discretionary 

1“ 

1“ 

4.2  Level  of  Analysis 

Pooled 

n 

1” 

1“ 

Individual  Performance 

1“ 

Sequential 

r 

r 

r 

Team  Performance 

(7 

Reciprocal 

r 

r 

r 

Comments:  Roles  were  divided  as  follows:  (a)  The  pilot  was  in  charge  of  flying  the  aircraft;  (b)  the 


gunner  operated  the  weapon  systems,  which  included  selecting,  loading,  and  shooting 
various  ammunition;  and  (c)  the  radar  specialist  was  responsible  for  monitoring  and 
interpreting  radar  systems  containing  critical  enemy  information.  Team  members 
communicated  with  each  other  during  the  simulation  through  an  aircraft  cockpit  system 
consisting  of  interconnected  microphone-equipped  headphones.  There  was  no 
redundancy  in  role  functions  (e.g.,  only  the  pilot  could  fly  the  helicopter;  only  the 
gunner  could  select,  load,  and  shoot  ammunition;  and  only  the  radar  specialist  had 
access  to  enemy  radar  and  waypoint  information). The  task  was  highly  interdependent: 
To  perform  it  effectively,  team  members  had  to  work  together  closely.  There  was  no 
way  to  effectively  complete  the  task  without  the  integrated  contributions  of  all  three 
members.  A  good  plan  of  attack  aided  team  performance,  although  it  was  impossible 
to  plan  for  the  precise  nature  and  timing  of  the  challenges  that  the  teams  faced. 

The  complex  and  dynamic  nature  of  the  task  was  primarily  due  to  three  elements:  (a) 
roving  enemy  helicopters  attempting  to  shoot  down  Apaches,  (b)  the  unfamiliar  and 
variable  terrain  (e.g.,  valleys,  mountains,  plains),  and  (c)  unanticipated  enemy  surface- 
toair  missiles.  Successful  performance  depended  on  the  ability  of  team  members  to 
coordinate  their  activities,  primarily  through  the  exchange  of  mission-critical 
information,  so  as  to  kill  enemy  targets  and  avoid  being  killed  by  enemy  forces.  Teams 
received  real-time  verbal  feedback  as  to  whether  they  had  killed  a  particular  target, 
and  they  had  access  to  a  computer  screen  indicator  of  weapon  supply  levels. 

Reference(s):  Article:  Marks  MA,  Sabella  MJ,  Burke  CS,  et  al.  The  impact  of  cross-training  on  team 

effectiveness.  J  AppI  Psychol  2002;87:3-13. 

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=1 
1 91 621 3&dopt=Abstract 


A- 10 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


No.10 

Scenario  name:  Radar  monitoring  task 


Platform  name  (asterisk 

*Team  Event-Based  Adaptive  Multilevel  Simulation  (TEAMSim) 

if  reviewed  in  platform 
report): 

Level  of  activity 

Reievance 

Strategic 

r 

Highly  relevant 

Operational 

i;/ 

Relevant  ^ 

Tactical 

r 

Somewhat  relevant 

Purpose 

Training 

r 

Research 

w 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

117 

r 

r 

Team  mental  model 

r 

r 

1” 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

r 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

W 

r 

Fixed 

w 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

1“ 

15 

1” 

Co-located 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

15 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

Back-up  behavior 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

Cognitive 

r 

15 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

15 

r 

r 

Automation 

15 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

15 

r 

r 

4.2  Level  of  Analysis 

Pooled 

1“ 

1“ 

1“ 

Individual  Performance 

15 

Sequential 

r 

r 

r 

Team  Performance 

w 

Reciprocal 

r 

r 

r 

Comments:  As  a  team,  participants  were  responsibie  for  working  interdependentiy  to  identify 

contacts,  make  decisions,  and  prevent  perimeter  intrusions.  The  task  incorporated 
both  additive  and  discretionary  interdependencies  that  compiied  to  team  performance. 
Workioad  was  equaiized  across  sectors.  However,  the  task  was  designed  to 
unpredictably — but  systematicaiiy —  overload  team  members.  This  interdependence 
created  discretionary  opportunities  for  other  members  to  shift  their  priorities  and 
strategies,  coordinate  effort,  and  contribute  to  team  performance.  Although  collective 
effort  contributed  to  team  performance,  team  members  working  outside  their  primary 
sector  could  not  simultaneously  work  toward  accomplishing  individual  goals.  Thus, 
consistent  with  the  multiple-goal  model,  overloads  were  designed  to  prompt  resource 
allocation  choices  toward  team  or  individual  goals. 

Reference(s):  DeShon,  R.P.,  Kozlowski,  S.  W.  J.,  Schmidt,  A.  M.,  Milner,  K.  R.,  Wiechmann,  D. 

(2004).  A  multiple-goal,  multilevel  model  of  feedback  effects  on  the  regulation  of 
individual  and  team  performance.  Journal  of  Applied  Psychology,  89(6),  1035-1056. 
http://iopsych.msu.edu/DeShon/Papers/DeShon%20et%20al%20(2004)%20- 
%20Team%20regulatory%20processes.pdf 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-11 


HUMANS)Sr£  l/S 

-r, 


No.11 

Scenario  name: 

Reconnaissance 

Platform  name  (asterisk 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

r 

Operational 

r 

Relevant 

r 

Tactical 

P 

Somewhat  relevant 

w 

Purpose 

Training 

r 

Research 

[7 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

r 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

w 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

r 

r 

Co-located 

1^ 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

i;7 

r 

r 

Automation 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

[7 

Reciprocal 

r 

r 

r 

Comments:  Agent  based  modeling  and  Behavior  Action  Simulation  Platform  (BASP).  Not  a  human- 

in-the-loop  simulation  platform. 

Reference(s):  http://www.leastsquares.com/papers/mws2001  .pdf 


A- 12 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


HUMANS)Xr£.l/S 

No?i2 


Scenario  name: 

Hostage  Extraction 

Platform  name  (asterisk 

Team  Performance  Assessment  Technoiogy  (TPAT) 

if  reviewed  in  piatform 

report): 

Level  of  activity 

Relevance 

Strategic 

r 

Highly  relevant 

Operational 

R7 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

w 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

117 

n 

1“ 

Team  mental  model 

1” 

r 

Medium 

1“ 

1“ 

1“ 

Task  mental  model 

r 

17 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

17 

r 

r 

Implicit 

r 

r 

Explicit 

r 

17 

r 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

(7 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

r 

Hierarchical  network 

17 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

r 

Co-located 

17 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

1“ 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

[7 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

1“ 

1“ 

1“ 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

K7 

r 

r 

Automation 

K7 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r? 

r 

r 

4.2  Level  of  Analysis 

Pooled 

1“ 

1“ 

1“ 

Individual  Performance 

Sequential 

r 

r 

r 

Team  Performance 

(7 

Reciprocal 

r 

r 

r 

Comments;  Scoring  is  provided  within  time/event  matrices  (specificaiiy,  decisions  are  piotted  by 

time),  such  that  key  factors  such  as  pianning,  strategy  formation,  decision-making,  and 
adaptation  to  change  are  demonstrated  both  graphicaiiy  and  tabuiariy.  Fifty  such 
dependent  measures  are  inciuded  and  scoring  occurs  on-iine.  53  scores  are  provided 
at  the  end  of  a  session,  inciuding  ten  measures  of  process-reiated,  sociai 
psychoiogicai,  and  performance  factors:  decision-making,  pianning,  strategy 
deveiopment,  situationai  awareness,  initiative,  communication,  cohesion,  ieadership, 
task  difficuity,  and  task  performance.  In  the  graphic  presentation  of  resuits,  arrowed 
iines  of  various  coiors  indicate  various  situations.  For  exampie,  a  participant  who 
performed  an  action  to  faciiitate  future  pians,  a  participant  who  pianned  an  action  but 
negiected  to  foiiow  through,  and  a  participant  who  pianned  an  action  but  then  foiiowed 
a  different  course  wouid  aii  be  iiiustrated.  The  patterns  of  arrows  and  coiors  that 
emerge  reveai  the  type  of  pianning  that  took  piace  (e.g.,  successfui  or  unsuccessfui). 

Reference(s);  Swezey,  R.  W.,  Hutcheson,  T.  D.,  Swezey,  L.  L.  (2000).  Deveiopment  of  a  second- 

generation  computer-based  team  performance  assessment  technoiogy.  Internationai 
Journai  of  Cognitive  Ergonomics,  4,  163-170. 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-13 


HUMANS)Sr£.l/S 


No.13 

Scenario  name:  Anti-submarine  Warfare 

Platform  name  (asterisk  Hierarchical  Task  Analysis  (Teams)  -  HTA  (T) 

if  reviewed  in  platform 

report): 

Level  of  activity  Relevance 

strategic  I  Highly  relevant 

Operational  ^  Relevant 

Tactical  I”  Somewhat  relevant 

Purpose 

Training  | 

Research 

r 

17 

r 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

IS« 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

u 

r 

Explicit 

r 

17 

r 

1.2  Team  History 

Centralized  network 

17 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

17 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

17 

r 

r 

Implicit 

r 

r 

r 

Co-located 

r 

r 

r 

Explicit 

r 

17 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

1“ 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

1“ 

r 

Automation 

1“ 

Conjunctive 

r 

1“ 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

17 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments: 


HTA(T)  breaks  down  team  goals  into  sub-goals  and  then  determines  which  sub-goals 
can  be  reached  only  by  teamwork.  Assessment  is  facilitated  by  the  presence  of  video 
and  voice  recordings  of  team  and  control  room  operators.  Qualitative  assessment  is 
drawn  solely  from  observation  by  a  subject  matter  expert,  and  so  requires  much 
training  on  the  part  of  the  expert  and  is  naturally  rather  subjective.  Each  episode, 
transcribed  from  the  recordings,  is  listed  and  coded  with  a  classification  scheme 
consisting  of  five  categories  of  behaviors  (also  introducing  a  small  amount  of 
subjectivity).  Communication  behaviors  are  composed  of  sending  information  and 
receiving  information,  and  coordination  behaviors  are  composed  of  discussion, 
collaboration,  and  synchronization.  Points  are  then  assigned  based  on  whether  each 
required  behavior  was  performed  correctly,  partially  correctly,  or  incorrectly/omitted. 


Reference(s): 


Annett,  J.,  Cunningham,  D.,  &  Mathias-Jones,  P.  (2000).  A  method  for  measuring 
team  skills.  Ergonomics,  43,  1076-1094. 


A- 14 


Team  IModelling:  Scenarios  and  Computational  iVIodels  stems 


No.14 

Scenario  name:  Dyads  (a  pilot  and  co-pilot) 

Platform  name  (asterisk  Low-Fidelity  Aviation  Research  Methodology 


if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

r 

Operational 

r 

Relevant 

r 

Tactical 

Somewhat  relevant 

Purpose 

Training 

r 

Research 

[7 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r/ 

r 

Explicit 

r 

[7 

r 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

w 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

i;7 

r 

Co-located 

1^ 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

i;7 

r 

r 

Automation 

r 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments:  N/A 

Reference(s):  Bowers,  C.  A.,  Salas,  E.,  Prince,  C.,  &  Brannick,  M.  (1992).  Games  teams  play:  A 

method  for  investigating  team  coordination  and  performance.  Behavior  Research 
Methods,  Instruments,  &  Computers,  24,  503-506. 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-15 


HUMANS)Sr£.l/S 


No.15 

Scenario  name:  C3  environment  -  monitoring  &  prosecuting  of  incoming  targets  on  a  radar  screen 

Platform  name  (asterisk  Team  Performance  Assessment  Battery  (TPAB) 


if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

1“ 

Operational 

r 

Relevant 

1“ 

Tactical 

17 

Somewhat  relevant 

(7 

Purpose 

Training 

r 

Research 

17 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

117 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

1 

r 

r 

Implicit 

1“ 

1“ 

r 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

1“ 

n 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

(7 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

1“ 

n 

r 

Co-located 

17 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

1“ 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

(7 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

17 

r 

Cognitive 

r 

1“ 

117 

Time  pressure 

r 

r 

17 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

117 

r 

r 

Automation 

117 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments:  The  primary  process  under  study  by  this  tooi  is  team  decision-making.  The 

straightforward  design  of  synthetic  work  aiiows  for  iow  cost,  ease  of  measurement, 
and  maximum  experimentai  controi,  whiie  stiii  repiicating  reaiistic  amounts  of  cognitive 
demand  and  workioad  -  the  vigiiance  required  in  TPAB  is  equivaient  to  that  of  reai-  iife 
tasks.  The  primary  dependent  measure  is  reaction  time,  as  state  changes  indicated 
within  the  monitoring  component  necessitate  certain  responses.  A  coordination 
component  is  aiso  provided,  as  resources  and  actions  must  be  synchronized  properiy 
to  act  against  incoming  targets  and  this  must  be  done  concurrentiy  with  the  monitoring 
task.  Because  individuai  and  team  tasks  must  be  performed  simuitaneousiy,  as  weii, 
generaiizabiiity  of  the  task  to  reai-iife  is  quite  high.  Task  characteristics  such  as  work 
ioad  and  time  pressure  and  situationai  characteristics  such  as  uncertainty  can  aiso  be 
examined  within  TPAB.  Lastiy,  further  deveiopment  has  extended  the  originai 
configuration  to  aiiow  team  size  to  vary. 

Reference(s):  Bowers,  C.  A.,  Urban,  J.  M.,  &  Morgan,  B.  B.,  Jr.  (1992).  The  study  of  crew 

coordination  and  performance  in  hierarchicai  team  decision  making  (Report  No.  TR-92- 
01).  Oriando,  FL:  University  of  Centrai  Fiorida,  Team  Performance  Laboratory. 


A- 16 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


HUMANS)XrC.l/S 


No.16 

Scenario  name:  Seeking  to  discover  the  intent  of  targets  based  on  characteristics 

Piatform  name  (asterisk  Team  interactive  Decision  Exercise  for  Teams  incorporating  Distributed  Expertise 

if  reviewed  in  piatform  (TiDE2) 


report): 

Levei  of  activity 

Reievance 

strategic 

r 

Highly  relevant 

17 

Operational 

r 

Relevant 

r 

Tactical 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

Scenario  evaiuation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

7 

r 

1“ 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

1“ 

17 

Explicit 

r 

r 

17 

1.2  Team  History 

Centralized  network 

r 

1“ 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

17 

r 

r 

Hierarchical  network 

17 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

1“ 

r 

Co-located 

(7 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

1“ 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

(7 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

1“ 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

1“ 

Time  pressure 

17 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

17 

r 

1“ 

Automation 

17 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

r 

1“ 

4.2  Level  of  Analysis 

Pooled 

17 

r 

1“ 

Individual  Performance 

1“ 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments: 

TIDE2  is  another  alternative  for  the  study  of  decision-making  in  complex,  ambiguous 

environments  in  command  and  controi. 

Reference(s):  Hoiienbeck,  J.  R.,  Sego,  D.  J.,  iigen,  D.  R.,  &  Major,  D.  A.  (1991).  Team  interactive 

decision  exercise  for  teams  incorporating  distributed  expertise  (TiDE2):  A  program 
and  paradigm  for  team  research  (Tech.  Rep.  No.  91-1).  East  Lansing:  Michigan  State 
University. 

Hoiienbeck,  J.  R.,  iigen,  D.  R.,  Sego,  D.  J.,  Hediund,  J.,  Major,  D.  A.,  &  Phiiiips,  J. 
(1995).  The  muiti-ievei  theory  of  team  decision  making:  Decision  performance  in 
teams  incorporating  distributed  expertise.  Journai  of  Appiied  Psychoiogy,  80,  292-316. 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-17 


No.17 

Scenario  name:  a  metropolis  crisis  control  center. 

Platform  name  (asterisk  C3  Interactive  Task  for  identifying  Emerging  Situations  (CiTiES) 


if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

17 

Operational 

17 

Relevant 

1“ 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

w 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

17 

1“ 

1“ 

Team  mental  model 

1“ 

17 

r 

Medium 

1“ 

l~ 

1“ 

Task  mental  model 

r 

R 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

M 

r 

r 

Implicit 

1“ 

1“ 

1“ 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

1“ 

1“ 

1“ 

Ad  Hoc 

P 

r 

r 

Decentralized  network 

k 

r 

r 

Fixed 

r 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

w 

r 

r 

Implicit 

17 

r 

r 

Co-located 

r 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

17 

1“ 

2.1  Task  Complexity 

Feedback 

r 

w 

r 

Uncertainty 

w 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

1“ 

17 

[“ 

Resource  Allocation 

r 

w 

r 

Cognitive 

r 

17 

r 

Time  pressure 

W 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

17 

r 

r 

Automation 

17 

Conjunctive 

1“ 

i~ 

i~ 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

1“ 

i~ 

[“ 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

1“ 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments:  CITIES  has  received  praise  for  imitating  real-  life  environments  successfully.  Additional 


equipment  can  be  purchased  for  more  specific  study  of  particular  factors  of  interest 
(e.g.,  devices  that  determine  which  microphone  is  in  use  or  measure  heart  rate  of 
participants).  NeoCITIES  is  an  update  and  extension  of  the  original  CITIES  task. 

Reference(s):  Wellens,  A.  R.,  &  Ergener,  D.  (1988).  The  C.l.T.I.E.S.  game:  A  computer-based 

situation  assessment  task  for  studying  distributed  decision-making.  Simulation  & 
Games,  19,  304-327. 


A- 18 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


HUMANS)XrC.l/S 


No.18 

Scenario  name:  A  radar-like  target  classification  task 

Platform  name  (asterisk  Team  Argus 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic  l~ 

Highly  relevant 

1“ 

Operational  l~ 

Relevant 

1“ 

T  actical  ^ 

Purpose 

Training  |~ 

Research  I? 

Somewhat  relevant 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

ft/ 

Explicit 

r 

r 

17 

1 .2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

Fixed 

w 

r 

r 

Hierarchical  network 

r 

r 

r 

1 .3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

r 

r 

Co-located 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

ft/ 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

\t/ 

r 

r 

Automation 

ft/ 

Conjunctive 

r 

r 

r 

Self-Report 

17 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

ft/ 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments:  Argus  simulates  a  radar-like  target  classification  task.  It  was  developed  to  support 

research  in  measuring  and  modeling  cognitive  work  load.  Argus  is  used  in  both  single¬ 
subject  and  team  modes.  In  Team  Argus  we  are  interested  in  what  levels  of  task 
workload  are  required  to  make  the  team’s  decision  making  process  and  performance 
deteriorate  and  what  types  of  communication  protocols  and  decision  aids  can  facilitate 
team  performance  at  high  levels  of  task  workload. 


Reference(s): 


http://cat.mm.rpi.edU/cogworks/publications/1 31_Schoelles&Gray01_BRMIC. pdf 


http://interruptions.net/literature/Miller-CHI01-p79-miller.pdf 


http://www.rpi.edu/~grayw/pubs/papers/schoelles/ms-wdg.pdf 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-19 


HUMANS)Sr£.l/S 


No.19 

Scenario  name: 

Recovering  weapons  from  hidden  caches 

Platform  name  (asterisk 

Neverwinter  Night 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

Strategic 

r 

Highly  relevant 

r 

Operational 

r 

Relevant 

17 

Tactical 

w 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1 .1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

1” 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

Implicit 

r 

r 

(7 

Explicit 

r 

r 

17 

1 .2  Team  History 

Centralized  network 

17 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

Fixed 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physicai  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Impiicit 

r 

r 

17 

Co-iocated 

r 

r 

Explicit 

r 

r 

(7 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

17 

r 

r 

Automation 

17 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

R? 

Reciprocal 

r 

r 

r 

Comments;  This  game  was  selected  because  it  can  be  used  to  simulate  cooperative  team  tasks 

and  it  facilitates  scenario  authoring  and  customization.  The  built-in  game  editor  tools 
are  designed  to  allow  users  to  customize  the  size  and  contents  of  the  game  world, 
including  synthetic  character  behavior  and  the  creation  of  customized  items.  A  range 
of  elements  were  incorporated  allowing  for  the  examination  of  culture  on  (1 )  team 
organization,  including  designated  leader  and  leaderless  conditions,  (2)  preferences 
for  negotiation  styles,  (3)  willingness  to  engage  in  tasks  not  related  to  the  primary 
mission  and  the  impact  of  the  requesting  individual's  status,  and  (4)  response  to 
insults  as  moderated  by  insulter  social  status.  Performance  is  team-based,  with 
participants  able  to  increase  the  team  score  by  completing  tasks  which  have  rewards 
while  managing  costs  and  penalties. 


Reference(s): 


http://nwn.bioware.com/ 


http://www.informs-cs.org/wsc05papers/134.pdf 


http://www.dodsbir.net/sitis/view_pdf. asp?id=NATOWarren. doc 


A-20 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


No.20 

Scenario  name:  North  African  “insertion  from  the  sea” 

Platform  name  (asterisk  DDD-III  simuiator 


if  reviewed  in  piatform 
report): 

Level  of  activity 

strategic 

Operational 

Tactical 

Purpose 

Training 

Research 

r 

ttj 

r 

r 

[7 

Relevance 

Highly  relevant 

Relevant 

Somewhat  relevant 

K/ 

r 

r 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

Medium 

1:7 

r 

r 

Task  mental  model 

r 

r 

[7 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

r/ 

Explicit 

r 

r 

[7 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

w 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

r 

i;7 

Co-located 

1^ 

r 

r 

Explicit 

r 

r 

[7 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

[7 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

r? 

r 

r 

Automation 

Conjunctive 

1“ 

r 

r 

Self-Report 

17 

Disjunctive 

r 

r 

r 

Cbservers 

17 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments:  The  DDD-III  is  a  distributed  ciient-server  simuiation  that  provides  a  fiexibie  framework 


in  which  to  study  team  decision  making  and  performance  in  compiex  situations. 
Reference(s):  http://www.dodccrp.org/events/1999/1999CCRTS/pdf_fiies/track_1/016entin.pdf 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-21 


No.21 

Scenario  name:  Capture-the-flag  event  at  a  “Platoon  vs.  Platoon”  level. 

Platform  name  (asterisk  Neverwinter  Nights 


if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

r 

Operational 

It/ 

Relevant 

It/ 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

w 

Research 

r 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

W 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

Implicit 

r 

r 

r 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

[7 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

r 

Hierarchical  network 

[7 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

w 

r 

r 

Implicit 

r 

r 

It/ 

Co-located 

r 

r 

r 

Explicit 

r 

r 

[7 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

n 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

It/ 

r 

r 

Automation 

n 

Conjunctive 

r 

r 

r 

Self-Report 

W 

Disjunctive 

r 

r 

r 

Cbservers 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

[7 

Reciprocal 

r 

r 

r 

Comments:  The  NwN  “engine”  Is  very  versatile  and  Is  capable  of  supporting  a  wide  variety  of 

activities.  It  therefore  offers  some  of  the  versatility  of  a  custom  testbed.  Indeed,  the 
NwN  engine  is  versatile  enough  to  emulate  or  recreate  many  “simpler”  games  such  as 
chess,  Battleship,  and  Scudhunt. 

Reference(s):  http://openmap.bbn.eom/~thussain/publications/2005_HFES_paper.pdf 


A-22 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


HUMANS)SrC.l/S 


No.22 

Scenario  name:  Radar-based  ATC  Tasks 


Platform  name  (asterisk 
if  reviewed  in  piatform 
report): 

Level  of  activity 

strategic 

Operational 

Tactical 

Purpose 

Training 

Research 

*ATC  team  training  device 

r 

r 

W 

W 

r 

Relevance 

Highly  relevant 

Relevant 

Somewhat  relevant 

r 

r 

[7 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

17 

Explicit 

r 

r 

17 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

r 

r 

r 

Co-located 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r~ 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

17 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

17 

r 

r 

Automation 

17 

Conjunctive 

r 

r 

r 

Self-Report 

17 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

[7 

Sequential 

r 

r 

r 

Team  Performance 

[7 

Reciprocal 

r 

r 

r 

Comments:  N/A 

Reference(s):  http://www.hf.faa.gov/docs/508/docs/cami/00_25.pdf 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 
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HUMANS)Sr£.l/S 

'r,  ■  ■^•'•*'14 

N0I23 

Scenario  name:  Ensuring  that  launching  a  space  vehicle  will  not  endanger  the  general  populace 

Platform  name  (asterisk  Multi-agent  Operation  Range  Simulation  Environment  (MORSE) 


if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

1“ 

Highly  relevant 

n? 

Operational 

r 

Relevant 

1“ 

Tactical 

w 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

1^ 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

n? 

1“ 

1“ 

Team  mental  model 

1“ 

1“ 

1“ 

Medium 

1“ 

1“ 

1“ 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

r 

Explicit 

r 

r 

r 

1 .2  Team  History 

Centralized  network 

1“ 

1“ 

1“ 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

17 

r 

r 

Hierarchical  network 

r 

r 

r 

1 .3  Physical  Distribution 

3.3  Coordination 

Distributed 

17 

r 

r 

Implicit 

1“ 

1“ 

17 

Co-located 

r 

r 

r 

Explicit 

r 

r 

17 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

1“ 

1“ 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

17 

r 

Cognitive 

r 

1“ 

1“ 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

1“ 

1“ 

Automation 

1“ 

Conjunctive 

n 

1“ 

r 

Self-Report 

17 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

1“ 

1“ 

r 

4.2  Level  of  Analysis 

Pooled 

r 

1“ 

r 

Individual  Performance 

17 

Sequential 

r 

r 

r 

Team  Performance 

[7 

Reciprocal 

r 

r 

r 

Comments:  MORSE  (Multi-agent  Operation  Range  Simulation  Environment)  is  an  environment  for 

three  team  members,  encompassing  some  of  the  challenges  facing  Range  Operations 
at  KSC  (Kennedy  Space  Center). 

Reference(s):  http://www.cs.cmu.edU/~softagents/papers/rectenwald_michael_2003.1  .pdf 

http://www.cs.cmu.edu/~softagents/morse/JohnSycaraMar03.ppt 

http://www.cs.cmu.edu/~pscerri/papers/hcii_05.pdf 


A-24 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


No.24 

Scenario  name: 


working  together  to  find  Scud  iaunchers 


Platform  name  (asterisk 

SCUDHunt 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

Strategic 

r 

Highly  relevant 

It? 

Operational 

r 

Relevant 

r 

Tactical 

It? 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

It? 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

K/ 

r 

r 

Team  mental  model 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

i^ 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

r 

Explicit 

r 

r 

W 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

It? 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

It? 

r 

r 

Implicit 

r 

r 

r 

Co-located 

It? 

r 

r 

Explicit 

r 

r 

P 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

K? 

r 

r 

Automation 

K7 

Conjunctive 

r 

r 

r 

Self-Report 

IP' 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

It? 

Reciprocal 

r 

r 

r 

Comments:  SCUDHunt  is  an  internet-based  game  of  command  and  controi  deveioped  by  the 

Center  for  Navai  Anaiyses  (CNA)  and  ThoughtLink,  inc.SCUDHunt  was  designed  as 
an  experimentai  test  bed  for  SSA  (Shared  Situationai  Awareness)  research,  aiiowing 
us  to  obtain  an  objective  measure  of  SSA,  in  the  game  context,  and  statisticaiiy 
anaiyze  the  data.  The  game  appears  deceptiveiy  simpie.  in  fact,  it  enabies  the 
demonstration  of  very  interesting  team  and  individuai  behaviors,  iike  ieadership  and 
decision-making,  and  yieids  a  rich  set  of  data. To  piay  effectiveiy,  piayers  need  to 
coiiaborate  and  share  information.  No  piayer  can  win  by  themseives  -  it  takes  a 
combined  team  effort  to  effectiveiy  depioy  aii  sensors  and  correctiy  interpret  the 
sensor  resuits.  This  created  an  exceiient  environment  for  expioring  a  team’s  shared 
SSA,  as  weii  as  for  examining  how  the  experimentai  conditions  affect  the  team’s 
overaii  decision  quaiity  (defined  as  their  abiiity  to  iocate  aii  of  the  hidden  SCUDs). 

Reference(s):  http://www.scudhunt.com/ 

http://www.thoughtiink.eom/pubiications.htm#SCUD 

http://www.thoughtiink.com/fiies/pdf/i\/ietaAnaiysisWeb.pdf 

Yiwaawsystems^  Team  Modelling:  Scenarios  and  Computational  Models  A-25 


HUMANS)Sr£.l/S 


No.25 

Scenario  name: 

Naval  Air  Defense  Warfare 

Platform  name  (asterisk 

Wright  State  Aegis  Simulation  Platform  (WASP)/Team  Aegis 

Simulation  Platform 

if  reviewed  in  platform 

(TASP)  (an  extension  of  WASP) 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

r 

Operational 

It? 

Relevant 

It? 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

(7 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

U 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

i;? 

r 

Explicit 

r 

r 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

Hierarchical  network 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

w 

r 

r 

Implicit 

r 

[;/ 

r 

Co-located 

r 

r 

r 

Explicit 

r 

17 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

17 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

W 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

r 

r 

r 

Automation 

i;? 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments:  The  purpose  of  TASP  is  to  extend  the  existing  suite  of  tools  known  as  WASP  to  better 

enable  researchers  to  assess  and  model  human  performance  in  a  team 
environment. The  proposed  effort  will  extend  WASP  by  adding  more  tasks  and 
responsibilities  in  a  three-person  team  configuration,  incorporate  hooks  for  software 
models  of  each  role  to  be  able  to  interact  with  the  simulation  in  real  time,  incorporate 
hooks  for  diagnostic  algorithms  and  feedback  algorithms,  enable  automatic  collection 
of  performance  data  to  include  automatic  speech  recognition,  develop  diagnostic 
algorithms  consistent  with  the  Team  Dimensional  Training  methodology,  design  and 
develop  software  agents  to  identify  and  measure  team  time  windows. 


Reference(s): 


http://www2.ie.psu.edu/Rothrock/Research/HPAM/hpam/tasp.htm 

http://www2.ie.psu.edu/Rothrock/Research/HPAM/hpam/wasp.htm 


A-26 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


HUMANS)Src.l/S 

No!26 


Scenario  name:  Janus  wargame 

Platform  name  (asterisk  wargaming  simuiation 

if  reviewed  in  piatform 

report): 

Level  of  activity 

strategic 

Operational 

Tactical 

Purpose 

Training  I” 

Research 

Relevance 

Highly  relevant 

Relevant 

Somewhat  relevant 

r 

15 

r 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

15 

r 

Explicit 

r 

[7 

r 

1.2  Team  History 

Centralized  network 

15 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

Fixed 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

[7 

r 

r 

Implicit 

r 

r 

r 

Co-located 

r 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

[7 

r 

Uncertainty 

[7 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

r 

r 

r 

Automation 

r 

Conjunctive 

15 

r 

r 

Self-Report 

17 

Disjunctive 

r 

r 

r 

Cbservers 

17 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

[7 

Reciprocal 

r 

r 

r 

Comments:  N/A 

Reference(s):  http://www.dsto.defence.gov.au/publications/2504/DSTO-TR-1372.pdf 


Hwmmsy  stems 
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HUMANS)Sr£  l/S 

-r, 

NoTt 

Scenario  name:  Fire  Fighting 

Platform  name  (asterisk  Networked  Fire  Chief  (NFC) 


if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

r 

Operational 

It? 

Relevant 

17 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

w 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

17 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

l~ 

i~ 

i~ 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

[“ 

[“ 

17 

Explicit 

r 

r 

17 

1 .2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

17 

r 

r 

Hierarchical  network 

r 

17 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

[7 

r 

r 

Implicit 

r 

r 

17 

Co-located 

r 

r 

Explicit 

r 

r 

[7 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

1“ 

1“ 

2.1  Task  Complexity 

Feedback 

r 

17 

r 

Uncertainty 

17 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

17 

r 

Cognitive 

r 

17 

r 

Time  pressure 

r 

[7 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

17 

r 

r 

Automation 

17 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

r 

Reciprocal 

r 

r 

r 

It  should  be  noted  that  NFC  does  not  mirror  everything  that  occurs  in  a  military  context. 
Rather,  it  is  a  tool  used  to  investigate  the  NDM  (Natural  Decision  Making)  theory.  It 
brings  the  sort  of  variables  into  play  that  would  be  involved  in  NDM  such  as 
uncertainty,  complexity,  and  feedback  loops. 

Chapman,  T.  (2000).  The  effect  of  management  structure  and  communication 
architecture  on  naturalistic  decision  making  performance.  University  of  Adelaide: 
Unpublished  Honours  Thesis. 


Comments: 


Reference(s): 


A-28 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


HUMANS)Xr£.l/S 


No.28 


Scenario  name: 

Fire  fighting 

Platform  name  (asterisk 

C3Fire 

if  reviewed  in  piatform 

report): 

Level  of  activity 

Relevance 

Strategic 

r 

Highly  relevant 

15 

Operational 

11/ 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

V/ 

Research 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

15 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

17 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

Implicit 

r 

r 

15 

Explicit 

r 

r 

(7 

1 .2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

[7 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

w 

r 

r 

Implicit 

r 

r 

15 

Co-located 

r 

r 

r 

Explicit 

r 

r 

(7 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

k 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

15 

r 

r 

Time  pressure 

W 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

15 

r 

r 

Automation 

15 

Conjunctive 

r 

r 

r 

Self-Report 

Disjunctive 

r 

r 

r 

Observers 

R? 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

t5 

Sequential 

r 

r 

r 

Team  Performance 

Reciprocal 

r 

r 

r 

Comments:  C3Fire  make  it  possible  to  configure  and  simuiate  different  forms  of  organizations  and 

ways  of  how  the  system  ailows  the  subjects  to  handie  and  to  exchange  information. 
Accordingiy,  it  is  possibie  to  accompiish  a  command  situation  that  have  comparabie 
simiiarities  to  current  or  envisioned  where  researchers  can  investigate  as  e.g.  team 
performance  from  a  predetermined  and  controiied  situation,  which  in  turn  couid  have 
different  ieveis  of  constraints  in  information  fiow.The  C3Fire  micro-world  can  be 
viewed  as  a  command,  control  and  communication  simuiation  environment,  which  can 
be  used  for  investigation,  experimentation  and  training  on  TDM  (team  decision 
making)  and  TSA  (team  situation  awareness). The  C3Fire  micro-worid  has  shown  to 
provide  an  exceiient  support  for  quantitative  data  retrievai,  which  in  turn  is 
suppiemented  with  quaiitative  data  retrievai  from  audio  and  video  recordings,  as  weli 
as  questionnaires. 


Reference(s):  http://www.c3fire.org/c3fire/home/home.en.shtmi 


http://www.dodccrp.org/events/2002/CCRTS_Monterey/Tracks/pdf/103.PDF 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-29 


HUMANS)Sr£  l/S 


No.29 

Scenario  name:  joint  command  and  control 

Platform  name  (asterisk  DDD-III 
if  reviewed  in  platform 
report): 

Level  of  activity 

strategic  I~ 

Operational  ^ 

Tactical 

Purpose 

Training  I~ 

Research 

Relevance 

Highly  relevant 

Relevant 

Somewhat  relevant 

k 

r 

r 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

k 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

k 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

k 

r 

r 

Implicit 

r 

r 

k 

Explicit 

r 

r 

k 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

k 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

k 

r 

r 

Implicit 

r 

r 

k 

Co-located 

r 

r 

r 

Explicit 

r 

r 

k 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

k 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

k 

r 

r 

Automation 

k 

Conjunctive 

r 

r 

r 

Self-Report 

k 

Disjunctive 

r 

r 

r 

Cbservers 

k 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

k 

r 

r 

Team  Performance 

k 

Reciprocal 

r 

r 

r 

Comments: 


Reference(s): 


Four  categories  of  data  were  collected  during  each  of  the  experiments:  1 .  DDD 
simulator-collected  data  or  MTWS  data  files  2.  Video  and  audio  tapes  3.  Observer- 
collected  data  4.  Player  self-report  data 

http://www.dodccrp.org/events/1999/1999CCRTS/pdf_files/track_1/103wolle.pdf 


A-30 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


No.30 

Scenario  name:  Airborne  Warning  and  Controi  System  (AW ACS)  Weapons  Director  Teams 


Platform  name  (asterisk 

‘DDDnet 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

It/ 

Highly  relevant 

17 

Operational 

i;/ 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

w 

Research 

Scenario  evaiuation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

1” 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

W 

r 

r 

3.2  Communication 

Teams  of  teams 

k/ 

r 

r 

Implicit 

r 

17 

r 

Explicit 

r 

IV 

r 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

IV 

r 

Fixed 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

I? 

r 

r 

Implicit 

r 

17 

r 

Co-located 

r 

r 

r 

Explicit 

r 

IV 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

IV 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

IV 

r 

Cognitive 

[“ 

1“ 

1“ 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

r 

r 

r 

Automation 

17 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

IV 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

IV 

Reciprocal 

r 

r 

r 

Comments:  The  DDDnet  is  an  internet-ready  version  of  a  Linux-based  coiiaborative  gaming  space 

that  connects  piayers  to  each  other  and  to  others,  such  as  observers,  confederates, 
trainers,  or  researchers,  in  the  DDDnet  observers  at  any  location  in  the  network  are 
able  to  observe  the  scenario  play  in  real  time.  They  can  view  the  screen  display  and 
electronic  communications  of  any  player,  and  communicate  to  one  another  via  email  or 
voice.  In  addition,  the  DDDnet  can  connect  players  to  one  another  for  interactive 
mission  planning,  debriefings  and  after-action  reviews.  DDD  simulations  in  general  are 
based  on  broad  command  and  control  (C2)  functions  and  have  been  demonstrated  to 
elicit  important  team-oriented  cognitive  processes  such  as  communication  and 
coordination,  resource  allocation  and  sharing,  and  decision  making. 


Reference(s):  Barnes,  C.,  Elliott,  L.,  &  Entin,  E.  (2001).  Employing  Internet2  Technology  to  Enable 

Collaborative  Research  and  Distributed  Training  in  Complex  Multi-Operation  Settings. 
Webnet  Journal,  October-December. 

http://dodccrp.org/events/2002/CCRTS_Monterey/Tracks/pdf/044.PDF 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-31 


HUMANS)Sr£.l/S 

'r.  . 

Na31 


Scenario  name: 

Airborne  Warning  and  Controi  System 

Platform  name  (asterisk 

C3STARS 

if  reviewed  in  piatform 

report): 

Level  of  activity 

Relevance 

strategic 

1“ 

Highly  relevant 

l~ 

Operational 

n? 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

W 

Purpose 

Training 

w 

Research 

w 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

[“ 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

It? 

Explicit 

r 

r 

p 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

1“ 

1“ 

1“ 

Co-located 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

1“ 

1“ 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

[“ 

1“ 

1“ 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

It/ 

1“ 

r 

Automation 

It/ 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

w 

Discretionary 

r 

[“ 

1“ 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

W 

Reciprocal 

r 

r 

r 

C3STARS  facility  offers  the  opportunity  to  investigate  complex  decision 
making  among  interdependent  team  members  within  a  dynamic  and  realistic  setting. 
Closed  circuit  video  and  audio  stations  permit  experimenters  to  directiy  observe  team 
interactions  and  remotely  record  aii  communications  (computer,  visuai  and  audio)  for 
later  analysis.  The  unique  simulations  integrate  hardware  and  software  resources, 
data  coliection  and  analysis  systems,  verbai  communication  networks,  command  and 
control  scenarios  and  team  performance  measures. 
http://dodccrp.org/events/2002/CCRTS_IVlonterey/Tracks/pdf/044.PDF 


Comments: 


Reference(s): 


A-32 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


HUMANS)Src.l/S 


No.32 


Scenario  name: 

Joint  Task  Force  (JTF) 

Platform  name  (asterisk 

DDD-III 

if  reviewed  in  piatform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

rs? 

Operational 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

[7 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

Implicit 

r 

r 

Explicit 

r 

r 

17 

1.2  Team  History 

Centralized  network 

r 

r 

(S' 

Ad  Hoc 

[7 

r 

r 

Decentralized  network 

r 

r 

[7 

Fixed 

r 

r 

r 

Hierarchical  network 

r 

r 

[7 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

[7 

r 

r 

Implicit 

r 

r 

n? 

Co-located 

r 

r 

r 

Explicit 

r 

r 

[7 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

[7 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

r 

r 

Automation 

13' 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

Reciprocal 

r 

r 

r 

Comments:  The  focus  of  this  study  is  on  the  reiative  effectiveness  of  three  organizationai 

structures  in  the  conduct  of  a  simuiated  Joint  Task  Force  mission. Because  of  the 
focus  in  the  paper  on  the  roie  of  coordination,  the  major  mission  tasks  requiring 
muitipie  assets  wiii  be  one  focus  of  the  anaiysis. 

Reference(s):  http://www.dodccrp.org/events/1999/1999CCRTS/pdf_fiies/track_1/102hocev.pdf 


Hwmmsy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


A-33 


HUMANS)Sr£  l/S 

-r, 

No!33 

Scenario  name:  Pilots  (ATC) 

Platform  name  (asterisk  Microsoft  Flight  Simulator 


if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

i;7 

Operational 

It7 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

[7 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

Team  mental  model 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

[7 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

r 

r 

r/ 

Explicit 

r 

r 

[7 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

[7 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

[7 

r 

r 

Implicit 

r 

r 

r 

Co-located 

r 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

r 

r 

Cognitive 

r? 

r 

r 

Time  pressure 

w 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

i;? 

r 

r 

Automation 

r 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Cbservers 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

17 

Reciprocal 

r 

r 

r 

Comments:  Military  air  crews  flew  two  simulated  missions.  Independent  judges  provided 

evaluations  of  the  same  six  team  process  variables  in  both  scenarios. 

Reference(s):  Brannick,  M.  T.,  Prince,  A.,  Prince,  C.,  &  Salas,  E.  (1995).  The  measurement  of  team 

process.  Human  Factors,  37,  641-651. 


A-34 


Team  Modelling:  Scenarios  and  Computational  Models  stems 


HUMANSVSrC.l/S 


No.34 

Scenario  name:  Joint  Task  Force  Group  (JTFG) 

Platform  name  (asterisk  ODD  III 
if  reviewed  in  piatform 
report): 

Level  of  activity 

strategic  Ii7 

Operational 

Tactical 

Purpose 

Training  I~ 

Research  W 

Relevance 

Highly  relevant 

Relevant 

Somewhat  relevant 

l~ 

r 

R 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

1“ 

[“ 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

w 

r 

r 

Implicit 

r 

r 

r 

Explicit 

r 

r 

r 

1.2  Team  History 

Centralized  network 

r 

r 

vs 

Ad  Hoc 

R 

r 

r 

Decentralized  network 

r 

r 

R 

Fixed 

r 

r 

r 

Hierarchical  network 

r 

r 

R 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

w 

r 

r 

Implicit 

1“ 

1“ 

1“ 

Co-located 

r 

r 

r 

Explicit 

r 

r 

r 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

1“ 

[“ 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

w 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

R 

r 

Cognitive 

n7 

1“ 

1“ 

Time  pressure 

W 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

1“ 

r 

Automation 

vs 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

[“ 

1“ 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

R 

Reciprocal 

r 

r 

r 

Comments: 


Reference(s): 


The  software  environment  allows  one  to  perform  a  comparative  analysis  of  different 
organizations  for  various  (user-defined)  performance  measures,  and  to  quantify  the 
robustness  of  a  given  organizationai  design.  The  commander  (CJTFG)  sets  out  to 
devise  a  pian  for  the  mission  that  will  specify  aii  the  tasks  to  be  completed,  as  well  as 
analyze  the  decision-making  involved  and  stipulate  who  completes  what  task,  which 
resources  are  used  to  compiete  each  specific  task,  and  how  JTFG  wiil  coordinate  in 
order  to  guarantee  the  best  performance. 

http://dodccrp.org/events/1999/1999CCRTS/pdf_fiies/track_1/079patti.pdf 
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No.35 


Scenario  name: 

Team  C2  Task  under  sustained  operation  environment 

Platform  name  (asterisk 

*AEDGE™ 

(Agent  Enabled  Decision  Guide  Environment) 

if  reviewed  in  platform 

report): 

Level  of  activity 

Relevance 

strategic 

It/ 

Highly  relevant 

It/ 

Operational 

It/ 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

r 

Purpose 

Training 

r 

Research 

1^ 

Scenario  evaluation: 


Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

It/ 

r 

r 

Team  mental  model 

1“ 

1“ 

r 

Medium 

1“ 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

r 

r 

r 

Implicit 

1“ 

1“ 

It/ 

Explicit 

r 

r 

w 

1.2  Team  History 

Centralized  network 

1“ 

1“ 

r 

Ad  Hoc 

r 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

Hierarchical  network 

r 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

r 

r 

r 

Implicit 

1“ 

1“ 

It/ 

Co-located 

r 

r 

r 

Explicit 

r 

r 

w 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

1“ 

1“ 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

r 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

w 

r 

Cognitive 

It/ 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

It/ 

r 

r 

Automation 

It/ 

Conjunctive 

r 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

1“ 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

It/ 

Sequential 

r 

r 

r 

Team  Performance 

w 

Reciprocal 

r 

r 

r 

Comments:  N/A 

Reference(s):  http://doclccrp.Org/events/2002/7th_ICCRTS/Tracks/pdf/1 1 3.pdf 


Barnes,  C.,  Whitmore,  J.,  Elliott,  L.,  &  Harville,  D.  (2004)  Assessing  Complex  Team 
Performance  in  a  Sustained  Operations  Environment.  Journal  of  the  International  Test 
and  Evaluation  Association,  25,  39-44. 

Barnes,  C.,  Elliott,  L.  R.,  Coovert,  M.  D.,  &  Harville,  D.  (2004).  Effects  of  Fatigue  on 
Simulation-based  Team  Decision  Making  Performance.  Ergometrika,  4,  2-12. 
http://www.ergometrika.org/volume4/BarnesEtAI.htm. 
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No.36 

Scenario  name: 

International  humanitarian  assistance 

Platform  name  (asterisk 

N/A 

if  reviewed  in  piatform 

report): 

Level  of  activity 

Relevance 

strategic 

r 

Highly  relevant 

r 

Operational 

Relevant 

r 

Tactical 

r 

Somewhat  relevant 

k 

Purpose 

Training 

[7 

Research 

r 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

k 

r 

r 

Implicit 

r 

r 

k 

Explicit 

r 

r 

k 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

k 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

r 

r 

r 

Hierarchical  network 

k 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

k 

r 

r 

Implicit 

r 

r 

k 

Co-located 

r 

r 

r 

Explicit 

r 

r 

k 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

k 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

k 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Outcome 

Additive 

k 

r 

r 

Automation 

r 

Conjunctive 

1“ 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Observers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

k 

Reciprocal 

r 

r 

r 

Comments: 

N/A 

Reference(s): 

http://www.vcds.forces.gc.ca/dgsp/pubs/rep-pub/dda/scen/scen-3_e.asp 
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No.37 

Scenario  name:  PEACE  SUPPORT  OPERATIONS 

Platform  name  (asterisk  N/A 
if  reviewed  in  piatform 
report): 

Level  of  activity  Relevance 

strategic  I~  Highly  relevant 

Operational  ^  Relevant 

Tactical  Somewhat  relevant 

Purpose 

Training 

Research  I” 

r 

r 

k 

Scenario  evaluation: 

Indep 

Dep 

Unknown 

Indep 

Dep 

Unknown 

1  Team  Factors 

3  Team  Processes 

1.1  Team  Size 

3.1  Shared  Knowledge 

Small 

r 

r 

r 

Team  mental  model 

r 

r 

r 

Medium 

r 

r 

r 

Task  mental  model 

r 

r 

r 

Large 

r 

r 

r 

3.2  Communication 

Teams  of  teams 

k 

r 

r 

Implicit 

r 

r 

k 

Explicit 

r 

r 

k 

1.2  Team  History 

Centralized  network 

r 

r 

r 

Ad  Hoc 

k 

r 

r 

Decentralized  network 

r 

r 

r 

Fixed 

k 

r 

r 

Hierarchical  network 

k 

r 

r 

1.3  Physical  Distribution 

3.3  Coordination 

Distributed 

k 

r 

r 

Implicit 

r 

r 

k 

Co-located 

r 

r 

r 

Explicit 

r 

r 

k 

3.4  Team  Adaptibility 

2  Task  Factors 

Monitoring 

r 

r 

r 

2.1  Task  Complexity 

Feedback 

r 

r 

r 

Uncertainty 

k 

r 

r 

Back-up  behavior 

r 

r 

r 

2.2  Workload 

3.5  Planning 

Physical 

r 

r 

r 

Resource  Allocation 

r 

k 

r 

Cognitive 

r 

r 

r 

Time  pressure 

r 

r 

r 

4  Team  Measures 

2.3  Task  Interdependence 

4.1  Cutcome 

Additive 

k 

r 

r 

Automation 

r 

Conjunctive 

1“ 

r 

r 

Self-Report 

r 

Disjunctive 

r 

r 

r 

Cbservers 

r 

Discretionary 

r 

r 

r 

4.2  Level  of  Analysis 

Pooled 

r 

r 

r 

Individual  Performance 

r 

Sequential 

r 

r 

r 

Team  Performance 

k 

Reciprocal 

r 

r 

r 

Comments: 

Reference(s): 


N/A 

http://www.vcds.forces.gc.ca/dgsp/pubs/rep-pub/dda/scen/scen-9_e.asp 
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Annex  B:  Scenario  Plan  Presented  to  Scientific 
Authority 

B1.  Introduction 


This  document  outlines  seenario  requirements  and  two  potential  seenarios  that  can  be  used  in  condueting  new 
human-systems  integration  (HSI)  research  on  teams  and  to  define  the  approaches  for  subsequent  experimental 
and  modelling  work.  Specifieally,  the  doeument  deseribes  the  general  approaeh,  assumptions,  seenario 
requirements,  factors  that  will  allow  experimental  control,  factors  that  may  be  manipulated,  general  categories 
of  measures  of  performance  (MOPs),  and  the  identification  of  two  scenarios  that  meet  these  requirements. 

This  represents  the  first  step  in  developing  seenario(s)  for  team  experiments  and  modelling  that  are 
representative  of  the  targeted  context. 

B2.  Method 

B2.1  General  approach 

It  was  deeided  that  the  generation  of  a  scenario  for  HSI  researeh  on  teams  should  involve  leveraging 
knowledge  gained  from  the  survey  of  team  literature,  the  identification  of  team  platforms,  and  knowledge  of 
future  requirements  of  the  CF  that  are  relevant  to  team  research.  This  approach  will  allow  the  HSI®  team  to 
provide  a  starting  point  for  the  experimental  and  modelling  work  to  be  condueted  in  subsequent  years  of  the 
Applied  Researeh  Project  (ARP).  Using  concrete  examples  of  seenarios  based  on  a  realistie  eontext  with 
identifiable  units  and  personnel  supports  the  development  of  a  high  fidelity  experimental  platform  appropriate 
for  team  modelling. 

B2.2  Assumptions 

Based  on  direetion  provided  by  the  Scientific  Authority  (SA),  it  was  decided  that  the  scenario  will  be  at  the 
operational  level  and  eomprise  a  Joint,  Interagency  and  Interdisciplinary  Task  Force  (TF),  in  whieh  teams-of- 
teams  can  be  ad  hoe  or  fixed  as  well  as  distributed  or  eo-located.  To  increase  the  face  validity  of  teams-of- 
teams  in  a  Joint,  Interageney  and  Interdisciplinary  setting,  we  suggest  representing  large  size  teams 
performing  cognitive  work  to  aehieve  common  objectives. 

B2.3  Generation  of  scenario  requirements 

Based  on  the  results  of  the  review  of  team  literature,  capabilities  of  relevant  team  researeh  platforms  and 
eonsideration  of  future  requirements,  the  HSI®  team  identified  factors  that  will  define  the  seenario  eontext, 
faetors  that  will  allow  experimental  eontrol,  factors  that  may  be  manipulated,  and  potential  categories  of 
MOPs.  All  of  these  will  be  integrated  into  the  final  seenario. 
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Scenario  context 

As  a  first  step  to  developing  a  seenario,  the  Scientifie  Authority  (SA)  identified  factors  that  are  crucial  to  the 
vision  of  this  project.  The  following  factors  are  fundamental  and  will  define  the  overall  context  of  the 
scenario: 

Joint; 

Interagency; 

Interdisciplinary,  and 
Operational  level  activity. 

After  careful  examination,  HSI®  has  identified  other  factors  that  emerge  from  accommodating  these 
requirements.  A  team  composed  of  Joint  CF  units,  multiple  agencies,  and  personnel  performing  distinct  roles 
can  be  satisfied  by  selecting  a  scenario  that  allows  for  teams-of-teams.  As  a  result  of  multiple  small  teams 
contributing  to  a  single  overarching  team,  the  experiment  can  include  both  ad  hoc  and  fixed  teams,  as  well  as 
distributed  and  co-located  teams.  Because  of  their  nature,  teams-of-teams  will  likely  be  ad  hoc  (because  they 
are  called  together  only  as  required  for  a  particular  operation)  and  distributed  (because  individual  agencies 
work  individually  from  different  locations),  while  sub-teams  are  more  likely  to  be  fixed  (e.g.  Red  Cross)  and 
co-located.  In  addition,  setting  teams-of-teams  as  a  requirement  will  presume  a  scenario/experiment  that  can 
support  large  scale  teams  (i.e.  three  teams  of  three).  Lastly,  to  ensure  relevance  to  operational  CF  activities, 
HSI®  suggests  focusing  on  cognitive  team  work  in  a  Command  and  Control  (C2)  task. 

Factors  to  be  controlled  in  the  Scenario 

As  noted  above,  preliminary  discussion  with  the  SA,  a  review  of  current  literature  on  teams,  and  knowledge 
of  future  requirements  of  the  CF,  led  to  the  identification  of  important  concepts  for  developing  a  scenario  for 
team  research.  The  following  team  and  task  factors  have  been  recognized  by  HSI®  as  important  factors  to  be 
controlled  within  the  scenario. 

Team  heterogeneity; 

Task  interdependence,  and 

Primary  as  well  as  secondary  events/tasks  within  the  scenario. 

Upon  final  selection  of  a  scenario  and  agreement  with  the  SA,  the  aforementioned  factors  will  be  explored  to 
determine  how  they  can  be  varied  to  pursue  different  research  aims  or  whether  they  should  be  held  static. 

Factors  that  may  be  manipulated 

Factors  within  the  scenario  that  could  be  potentially  manipulated  during  experimentation  include: 

Time  pressure; 

Workload  (related  to  time  pressure); 

Secondary  tasks  (i.e.  noise); 

Resources  available; 

Communication,  and 
Level  of  uncertainty. 
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The  factors  that  will  ultimately  be  manipulated  during  experiments  with  teams  will  likely  be  determined  when 
the  experiment  is  defined,  however,  the  scenario  must  be  able  to  support  manipulation  of  these  factors. 

Categories  of  MOPs 

Although  specific  MOPs  will  be  defined  at  a  later  point,  three  categories  of  MOPs  that  could  be  appropriate 
include  shared  knowledge  (i.e.  mental  models),  communication  (e.g.  number  of  missed  communications)  and 
team  performance  measures  (e.g.  quality  of  plan). 


Summary  of  scenario  requirements 


Scenario  context 

Factors  that  allow 
experimental  control 

Factors  that  may  be 
manipulated 

Categories  of  MOPs 

Joint 

Team  heterogeneity 

Time  pressure 

Shared  knowledge 

Interagency 

Task  interdependence 

Workload 

Communication 

Interdisciplinary 

Primary  and  secondary 
eventsdasks 

Secondary  tasks 

Team  performance 
measures 

Operational  level  activity 

Resources  available 

Communication 

Level  of  uncertainty 

B2.4  Identification  of  potential  scenarios 

At  this  point  two  potential  scenarios  have  been  identified,  however  these  scenarios  can  be  tweaked  or  more 
scenarios  can  be  generated  depending  on  further  direction  from  the  SA.  These  two  scenarios  are  described  in 
detail  below.  The  first  scenario  is  a  Domestic  Operation  (DOMOPS)  to  address  the  Winnipeg  Floods.  The 
second  scenario  is  also  based  on  a  real-life  operation  that  focuses  on  an  international  peace  support  operation 
involving  non-combatant  evacuation  with  a  multinational  joint  forces  headquarters  (HQ)  responsible  for 
mission  planning.  It  is  believed  that  the  use  of  scenarios  based  on  real-life  operations  will  prove  more  fruitful 
for  modeling  future  Canadian  Force  (CF)  activities  and  will  also  improve  the  face  validity  of  the  scenarios  for 
CF  personnel. 

For  each  of  the  two  potential  scenarios,  the  following  were  identified: 

1.  What  entities  are  involved?  (i.e.  military,  navy,  air  force,  OGD,  NGO  etc.)  Who  are  the  team 
member’s? 

2.  How  are  teams  organized? 

3.  At  what  level  of  activity  does  the  team  perform?  What  function  does  the  team  perform?  (i.e. 
strategic,  operational,  tactical) 

4.  What  are  the  team’s  goals  and  priorities? 

5.  What  are  the  primary  and  secondary  tasks? 

6.  What  demands  are  imposed  on  the  team  in  typical  situations  and  in  emergency  situations? 

7.  What  is  the  physical  distribution  of  teams? 

8.  Is  the  scenario  realistic? 
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It  is  anticipated  that  one  of  the  seenarios  presented  in  this  report  will  be  seleeted  for  the  purpose  of  modelling 
teams,  and  that  the  scenario  will  provide  the  foundation  for  subsequent  work  to  be  conducted  for  this  ARP. 
However,  if  either  of  these  scenarios  does  not  meet  the  requirements  of  the  project,  an  alternative  scenario  ean 
be  created. 

Table  B1:  Scenarios:  domestic  joint  force  operation  &  international  peace  support  operation 


Points  of  interest 

Scenario:  Domestic  joint  force 
operation  -  Winnipeg  Fioods  (OP 
ASSiSTANCE) 

Scenario:  internationai  peace  support 
operation 

Entities  invoived  (i.e.  miiitary, 
navy,  airforce,  OGD,  NGO 
etc.)  and  team  members 

Nationai  and  Provinciai/Municipai: 

C;fy  Departments -Ambuiance,  Fire, 

City  of  Winnipeg  Poiice  Service, 

Harbour  Patroi,  Sociai  Services, 
Emergency  Preparedness  and 
Coordination  Committee  (EPCC) 
A/af/ona/-  RCMP,  Emergency 
Preparedness  Canada  (EPC) 

Outside  Agencies  -  Media 

Pubiic  Sector-  Emergency  Pubiic 
Information  Team  (EPIT),  Pubiic  Heaith, 
Pubiic  Utiiities,  Pubiic  Works 

Voiunteer  Groups  -  United  Way,  Meais 
on  Wheeis,  Red  Cross,  Saivation  Army 

Department  of  Nationai  Defence: 

Joint  Task  Force,  8,500  CF  personnei, 
2,850  vehicies,  131  water  craft  and  34 
aircraft 

United  Nations  observers  on  the  ground 

Canadian  government  officiais  at  the 
Canadian  Embassy 

Canada  has  been  approached  by  the 
United  States  and  Britain  to  iead  a  muiti- 
nationai  joint  task  force 

Joint 

Organization  of  teams 

Teams-of-teams  ied  by  Joint  Task  Force 
Headquarters  (JTFHO) 

Teams-of-teams  -  Assigned  units  wiii 
operate  under  CPCCN  of  the  Canadian 
Task  Force  Commander  but  CPCCM 
wiii  remain  with  the  respective  nations. 
Forces  wiii  act  under  nationai  Ruies  of 
Engagement,  coordinated  through  the 
Canadian  Commander. 

Levei  of  activity/function  of 
team  (strategic,  operationai, 
tacticai) 

Strategic,  Cperationai  and  Tacticai 

Cperationai  and  Tacticai 

Team  goais  and  priorities 

To  evacuate  and  accommodate  aii 
those  in  danger  and  minimize  property 
damage 

Safe  extraction  of  Canadian,  American 
and  British  citizens  from  Caribba. 

Primary  and  secondary  tasks 

Primary  Tasks: 

Evacuate  peopie  in  danger  zones 

Ensure  open  iines  of  communication 
with  citizens 

Accommodation  for  those  evacuated 
Protection  of  high  risk  properties 

Dike  construction 

Primary  Tasks: 

In  cooperation  with  American  and  British 
miiitary  pianners  determine  the  optimai 
task  force  composition. 

Determine  the  operationai  command 
and  controi  structure 

Secondary  Tasks: 
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Points  of  interest 

Scenario:  Domestic  joint  force 
operation  -  Winnipeg  Fioods  (OP 
ASSiSTANCE) 

Scenario:  internationai  peace  support 
operation 

Sandbags 

Secondary: 

Deter  crime 

Protection  of  properties  at  medium  and 
iow  risk 

The  extent  of  human  suffering  due  to 
instabiiity  foiiowing  the  eiection  may  be 
such  that  the  United  Nations  wouid  ask 
for  assistance  to  secure  the  reiative 
safety  of  major  pockets  of  the 
population.  It  is  Canada's  current 
position  that  it  will  not  interfere  with  the 
internal  affairs  of  Caribba  unless  it  can 
be  clearly  demonstrated  that  the 
government  is  systematically  acting 
against  its  own  population. 

Demands  imposed  on  the 
team  in  typicai  situations  and 
in  emergency  situations 

High  time  pressure 

Infrastructure  (i.e.  compiimentary 
networks) 

Low  ievei  of  uncertainty 

High  Time  pressure 

High  Levei  of  uncertainty: 

It  is  impossible  to  predict  what  the 
authority  of  the  regime  will  be  following 
the  election. 

The  potential  presence  of  larger 
numbers  of  Caribbanian  civilians  has  to 
be  considered  in  planning. 

Physicai  distribution  of  teams 

Distributed 

Distributed 

Reaiism  of  scenario 

Based  on  a  historicai  event  that  was 
successfuiiy  bandied  but  couid  be 
improved 

Fictitious,  but  comprising  elements  of 
real  events 

B3  Next  steps 

Once  a  decision  has  been  reached  regarding  the  factors  to  include  in  the  final  scenario,  the  following  work 
must  be  performed  to  create  a  comprehensive  scenario  that  will  support  HSI  team  research  and  modelling: 

1.  Identify  clear  and  meaningful  start  and  end  points  for  the  scenario; 

2.  Create  of  a  representative  timeline  of  relevant  events  and  activities; 

3.  Highlight  key  decisions  and/or  action  points  for  team  members; 

4.  Define  positive  versus  negative  outcomes; 

5.  Support  the  identification  of  one  or  more  HSI  intervention(s)  whose  effectiveness  may  be  explored  in  a 
future  model  or  experiment; 

6.  Propose  hardware  and  software  requirements  for  an  experimental  platform  appropriate  for  an 
experiment  using  the  scenario; 

7.  Further  develop  MOPs  and  Measures  of  Effectiveness  (MOEs),  and 

8.  Propose  one  or  more  augmentation(s)  or  variation(s)  to  the  scenario  that  can  be  used  for  training 
subjects  for  team  experiment(s)  or  for  testing  the  generalizability  of  the  team  model. 

A  final  report  encompassing  the  scenario  development  and  tool  evaluation  will  be  submitted  to  the  SA. 
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Annex  C:  Modified  Scenario  Plan 


Development  of  Team  Seenario  -  Guidanee  on  Next  Steps 
Target:  Teams-of-teams  in  joint,  interageney  and  distributed  environment 
team  1:  DND  -  higher  level  of  eommand  (e.g.,  CanadaCOM) 
team  2:  DND  -  lower  level  of  eommand  (e.g.,  JTFA,  JTFC,  ...) 
team  3:  OGD  (e.g.,  PSEPC) 

Aim:  To  explore  eollaboration  and  eoordination  that  takes  plaee  vertieally  (e.g.,  teams  1&2)  and 
horizontally  (e.g.,  teams  1&3),  and  possibly  within-  and  aeross-  teams.  To  support  low  to  medium  fidelity 
experiments  and  eomputational  modelling,  to  examine  faetors  sueh  as:  team  strueture,  role  assignment, 
eommunieation,  integration,  leadership,  eommon  intent,  readiness 

Context:  Domestie  operations 

Approaeh:  Consider  multiple  seenarios  that  will  require  eollaboration  between  the  same  3  nodes,  and 
where  DND  will  play  a  substantial  rather  than  peripheral  role. 

Aetion  Items: 

Define  multiple  (instead  of  one)  seenarios,  but  with  less  granularity,  sketehed  out  rather  than  fully 
instantiated,  for  example: 

seenario  1:  natural  disaster  (e.g.,  Winnipeg  floods  would  be  aeeeptable) 
seenario  2:  terrorist  (e.g.,  radiologieal  threat) 
seenario  3:  pandemie  (e.g.,  avian  flu,  SARS) 

Basieally,  for  eaeh  seenario,  answer  a  modified  set  of  questions  (ef.,  page  5  in  your  plan  for  team 
modelling  seenario): 

1 .  (edited)  What  3  entities/teams  are  involved?  Who  are  the  team  members  within  eaeh  entity? 

2.  (edited)  Flow  are  teams  organized? 

3 .  Flow  is  responsibility  shared/distributed? 

4.  Flow  is  authority  shared/distributed? 

5.  (edited)  What  funetion  does  eaeh  team  perform?  (i.e.  strategie,  operational,  taetieal)? 

6.  (edited)  What  are  the  teams’  goals  and  priorities?  What  are  the  team  members’  goals  and 
priorities? 

7.  What  are  the  teams’  primary  and  seeondary  tasks?  What  are  the  team  members’  tasks? 

8.  What  demands  are  imposed  on  the  team  in  typieal  situations  and  in  emergeney  situations? 

9.  What  is  the  physieal  distribution  of  teams? 

10.  (omit) 

11.  (new)  Flow  do  the  teams  eommunieate  and  eoordinate?  (i.e.,  main  tools  and  proeesses) 

In  addition,  answer  the  following  questions  by  drawing  from  literature  review: 
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which  factors  on  team  performance  or  team  effeetiveness  ean  eaeh  seenario  explore? 

whieh  faetors  are  expeeted  to  be  espeeially  influential  in  whieh  seenarios?  would  the 
influential  faetors  be  the  same  or  different  aeross  seenarios? 

for  eaeh  influential  faetor,  what  level  of  the  faetor  does  eaeh  seenario  represent? 

How  well  does  eaeh  seenario  support  team  experiments? 

How  well  does  eaeh  seenario  support  eomputational  modelling  of  teams? 

Is  there  a  eoneeptual  model  of  team  performanee/effeetiveness  (from  literature)  that  ean  be 
tested  by  experiments  using  these  seenarios? 

Is  there  a  eomputational  model  of  team  performanee/effeetiveness  (from  survey)  that  ean  be 
tested  by  experiments  using  these  seenarios? 

To  eonduet  experiments  with  low/medium  fidelity  and  medium/high  eontrol,  who  should 
partieipate  in  the  experiments  (i.e.,  profile  sketeh)? 

What  sereening  /  training  should  be  provided  to  the  experiment  partieipants? 

In  terms  of  Next  Steps  (ef.,  page  8  in  your  plan  for  team  modelling  seenario): 

1.  (de-emphasize)  dentify  elear  and  meaningful  start  and  end  points  for  the  seenario;  -  >  de- 
emphasize 

2.  (de-emphasize)  Create  of  a  representative  timeline  of  relevant  events  and  aetivities  ->  de- 
emphasize 

3.  (important)  Highlight  key  deeisions  and/or  aetion  points  for  team  members; 

4.  (important,  relates  to  #1)  Define  positive  versus  negative  outeomes; 

5.  (edited)  identify  one  or  more  HSI  intervention(s)  whose  effeetiveness  may  be  explored  using 
these  seenarios 

6.  (de-emphasize)  Propose  hardware  and  software  requirements  for  an  experimental  platform 
appropriate  for  an  experiment  using  the  seenario; 

7.  (important)  Further  develop  MOPs  and  Measures  of  Effeetiveness  (MOEs) 

8.  (omit) 
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Annex  D:  Questionnaire  Sent  to 
Computational  Model  Developer  Organization 


Dear  Sir/Madam, 

We  are  conducting  an  assessment  on  tools  for  modeling  team  processes  (computational  models) 
on  behalf  of  DRDC  (Defence  Research,  Defence  Canada).  We  are  very  interested  in  Brahms. 
You  are  highly  appreciated  if  you  are  kind  enough  to  take  a  moment  to  clarify  the  following 
questions  for  us  as  to  the  Brahms: 

Is  the  Brahms  capable  of  doing/supporting  the  following  functions? 

*Generally,  please  answer  YES  or  NO. 

*Please  also  give  a  brief  explanation  when  necessary. 

*Some  question  maybe  hard  or  not  applicable  to  answer.  You  can  just  ignore  them. 

1.  Measures  Workload? 

Brahms  includes  a  full  agent-  (BDI)  and  object-oriented  language;  you  can  program  any 
measurement  you  want  to  make  during  a  simulation. 

2.  Compatible  with  IPME? 

Brahms  has  an  agent-based  (BDI)  language  and  a  Java  API.  Any  human-performance  model 
could  be  interfaced  with  as  an  agent  in  Brahms. 

3.  Scenario  Flexibility? 

The  Brahms  language  allows  for  initial-beliefs  and  creation  of  agents  and  objects  dynamically. 
This  way  you  can  create  scenarios  for  simulation.  The  language  also  allows  the  creation  of  new 
beliefs  via  conclude  statements  that  can  have  certainty  factors.  This  enables  flexibility  for  Monte- 
Carlo  sims. 

4.  Domain  Independent? 

Yes,  Brahms  is  a  domain  independent  BDI  language. 

5.  Model  Team  as  Entity? 

Agents  can  represent  any  abstract  user-defined  role,  including  representing  an  entire  team. 

Agents  can  have  attributes  to  allow  the  creation  of  beliefs  about  the  agent  for  any  agent  or  object, 
and  facts  in  the  world  about  the  agent.  Furthermore,  you  can  define  inference  rules  (forward- 
chain  reasoning  and  situation-action  rules),  activities  and  activity-rules  (rules  that  execute 
activities,  take  time  and  thus  allow  for  representing  situated  action.  One  can  also  define  initial 
beliefs  for  the  agent  and  world  facts  about  the  agent.  Each  agent  has  a  belief-set  that  contains  all 
the  beliefs  of  the  agent  at  any  moment  in  time.  Agents  perform  activities  and  situated  action  by 
the  firing  of  inference  and  activity-rules  by  matching  their  preconditions  to  the  current  belief-set. 

6.  Model  Team  as  Group  of  Individuals? 

Yes,  Brahms  agents  can  be  members  of  one  or  more  Groups.  Groups  can  represent  any  abstract 
user-defined  role,  including  representing  an  entire  team.  A  Brahms  Group  can  define  a 
functional,  organizational,  social  group,  or  community  of  practice.  Agents  inherit  everything 
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defined  in  a  Group.  Anything  that  can  be  defined  at  the  agent-level  can  be  defined  at  the  Group- 
level  and  is  inherited  by  the  agents  that  are  a  member  of  the  Group. 

7.  Model  Specified  Team  Tasks? 

Any  task  can  be  modeled  as  a  set  of  activities  (or  as  a  composite  activities  which  is  decomposed 
into  a  set  of  inference  rules  (called  thoughtframes)  and  sub-activities  and  their  activity-rules 
(called  workframes).  Team  tasks  can  thus  be  modeled  at  the  Group  level,  or  at  the  Agent  level. 
This  is  up  to  the  modeler. 

8.  Model  Specified  Team  Interactions? 

A  predefined  type  of  activity  is  a  "communicate  activity" .  In  communicate  activity  the  agent  can 
communicate  and/or  receive  beliefs  to/from  other  agents  or  objects.  Using  a  combination  of 
communicate  activities  we  can  model  generic  Team  interactions  (such  as  information  exchange  in 
a  meeting),  or  a  specific  interaction  with  objects  (e.g.  Typing  in  your  pin-code  at  an  ATM 
machine),  or  a  specific  communication  that  always  happens  (e.g.  I  always  say  my  name  when  I 
pick  up  the  telephone). 

9.  Model  HSI  (Human  System  Integration)  Interventions  of  Interest? 

Since  Brahms  also  models  objects,  we  can  model  systems  as  objects  (not  just  agents).  We  can 
model  human-system  interaction  very  easily  and  naturally.  More  specifically,  since  any  agent  can 
be  modeled  as  a  Java  agent,  we  can  easily  create  an  interface  to  the  actual  system. 

10.  Analyze  Team's  Strategies? 

Since  Brahms  is  an  agent  language  and  you  can  easily  model  teams  and  team  tasks,  you  can 
easily  analyze  team  strategies  by  analyzing  the  creation,  communication  and  changing  of  agent 
beliefs  as  they  are  performing  work  activities. 

11.  Analyze  Team's  Performance? 

See  answer  for  #10.  Activities  can  easily  be  modeled  as  creation  and  work  on  objects,  which  can 
be  easily  used  to  calculate  individual  agent  performance  and  aggregate  this  up  to  team 
performance. 

12.  Available  in  Public  Domain? 

Yes.  Brahms  can  be  downloaded  and  used  for  free.  However,  Brahms  is  not  Open  Source. 
Brahms  includes  an  agent-based  language,  compiler,  and  virtual  machine/simulation  engine. 
Interactive  Development  Environment  (the  Composer  and  soon  a  Brahms  plug-in  for  Eclipse) . 

13.  Stable? 

Brahms  has  been  stable  for  more  than  five  years,  although  we  are  continuing  adding  new 
features. 

14.  Real-Time  Computer  Generated  Forces? 

Brahms  can  easily  be  used  for  real-time  applications,  including  CGE.  Brahms  has  been  used  at 
NASA  for  the  development  of  a  multi-agent  workflow  environment,  creating  personal  agents  for 
astronauts  and  robots.  We  have  integrated  speech  dialog,  GUIs  and  planning  systems  as  human- 
agent  interfaces  to  Brahms  agents 
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No. 

Scenario  name 

Platform  Name 

Relevance 

Scenario 

1 

Search-and-Rescue  Mission  in 
Antarctica 

NASA  Ames  Centre  - 
Distributed  Research 
Facilities  (NASA) 

Highly  Relevant 

Space  mission:  A  computer-based  simulated  search  and  rescue 
mission  set  in  Antarctica  was  developed  to  study  team  interaction  and 
decision  making  performance.  Four-member  teams  must  work 
together  to  locate  a  lost  party  sent  to  repair  a  malfunctioning 
communication  antenna.  Teams  must  develop  plans  and  strategies, 
share  information,  manage  resources,  and  cope  with  unexpected 
problems  under  time  pressure.  Both  task  and  team  stressors  are 
manipulated  to  induce  cognitive  and  emotional  arousal.  Task 
performance,  physiological  measures  (ECG,  respiration,  SCL,  EMG, 
PPG),  voice  and  email  communication,  personality,  team  dynamics, 
and  facial  affect  measures  are  being  analyzed  to  identify  the  relations 
between  stress,  team  interaction,  and  team  performance. 

2 

Team  Resource  Allocation  Problem 
in  Emergency  Crisis  Management 

NeoCITIES 
(Pennsylvania  State 
University  and 

Purdue  University) 

Highly  Relevant 

The  simulation  is  designed  to  conduct  empirical  research  on  team 
cognition  and  decision-making  within  a  distributed  environment. 
NEOCITIES  is  an  interactive  computer  program  designed  to  display 
information  pertaining  to  events  and  occurrences  in  a  virtual  city 
space.  The  teams  in  the  simulation  represent  three  separate 
services  (Police,  Fire/EMS,  and  Hazmat)  in  which  they  must  assess 
situations,  interact  and  communicate  according  to  their  inter-team 
and  intra-team  roles,  allocate  resources  in  a  timely  manner,  and 
make  decisions  within  the  context  of  emergency  crisis  management. 
Scenarios  used  could  address  isolated  and  mundane  events, 
inclusive  events  that  have  the  potential  to  escalate  in  complexity, 
potential  terrorist  activities  (international  or  domestic). 
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No. 

Scenario  name 

Platform  Name 

Relevance 

Scenario 

3 

Uninhabited  Air  Vehicle  (UAV) 

Ground  Control  Operations  to 

Taking  Reconnaissance  Photos 

Synthetic  Task 
Environment  (STE)  in 
Cognitive 

Engineering 

Research  on  Team 
Tasks  (CERTT)  Lab 

Relevant 

Simulation  of  Uninhabited  Air  Vehicle  (UAV).  The  task  is  based  on 
the  actual  operations  of  the  Air  Force's  Predator  UAV  (UAV  for  the 
purpose  of  taking  reconnaissance  photos).  Teams  of  three  members 
have  distinct  roles.  The  Air  Vehicle  Operator  controls  airspeed, 
altitude,  heading,  and  monitors  the  UAV  systems.  The  Payload 
Operator  adjusts  camera  settings  to  take  target  photos  and  monitors 
camera  equipment.  The  Data  Exploitation,  Mission  Planning  and 
Communications  Operator  oversees  the  mission,  plans  a  route  under 
various  constraints,  and  reports  locations  and  restrictions. 

4 

Command  and  Control  Simulation  - 
Defending  a  region  against  invasion 
from  unfriendly  entity 

DDD 

Highly  Relevant 

The  DDD  is  a  dynamic  command  and  control  simulation  requiring 
team  members  to  monitor  activity  in  a  geographic  region  and  defend 
it  against  invasion  from  unfriendly  ground  or  air  tracks  that  enter  that 
region.  Participants  were  seated  in  close  proximity  to  one  another  at 
four  networked  computer  terminals.  Verbal  communication  was  the 
only  method  of  communication  allowed  during  the  task.  Team 
members  were  free  to  talk  as  much  or  as  little  as  they  wanted.  The 
geographic  region  is  partitioned  into  four  quadrants  of  equal  size. 

Each  team  member  is  assigned  the  responsibility  for  one  of  the  four 
quadrants.  The  objective  of  the  simulation  is  to  identify  tracks  that 
enter  the  space,  determine  whether  they  are  friendly  or  unfriendly, 
and  if  unfriendly,  keep  them  out  of  the  restricted  zones. 

5 

Combat  Information  Center  - 
Monitoring  radar  display  for  targets 

Tactical  Navy 

Decision  Making 
System  (TANDEM) 

Relevant 

In  TANDEM,  subjects  perform  a  sequence  of  time  critical  information 
gathering  and  communications  tasks  to  identify  targets  then  decide 
whether  to  shoot  or  clear  each  target.  TANDEM  is  a  simulated  radar 
display  that  spots  a  predetermined  number  of  targets  on  the  screen. 
The  task,  in  essence,  is  to  determine  the  type  and  intent  of  the  target, 
and  take  appropriate  action. 
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No. 

Scenario  name 

Platform  Name 

Relevance 

Scenario 

6 

Naval  Surveillance  and  Threat 
Assessment  Operation 

Team  and  Individual 
Tactical  Assessment 
Network  (TITAN) 

Relevant 

Four  team  members  are  asked  to  imagine  themselves  as  officers 
aboard  a  naval  ship.  Their  mission  is  to  evaluate  the  threat  posed  by 
the  air,  surface  and  subsurface  traffic  (aka.  'contacts')  in  their  ship's 
vicinity.  Their  ship  and  the  contacts  surrounding  it  are  displayed  on 
the  radar  screen  at  each  workstation.  The  team's  task  begins  with 
the  Leader  selecting  a  contact  for  the  subordinates  to  evaluate.  The 
Leader  waits  for  each  subordinate  to  use  the  information  gathered  by 
the  ship's  sensors  to  evaluate  the  threat  level  of  the  contact.  Upon 
reviewing  their  respective  contact  information  the  subordinates  each 
submit  a  threat  assessment  to  the  Leader.  Once  the  Leader  receives 
all  three  threat  assessments  (s)he  synthesizes  the  information  and 
submits  a  final  threat  assessment  on  behalf  of  the  team.  The  team 
will  later  be  able  to  review  feedback. 

7 

Tank  Battle  Game 

Bolo 

Somewhat 

Relevant 

Team  members  were  seated  side  by  side  at  three  computers,  and 
each  controlled  an  on-screen  'tank'  and  worked  in  a  computerized 
alliance  with  fellow  team  members.  Teams'  targets  in  this  exercise 
were  sixteen  enemy  'pillboxes'  that  would  attack  and  attempt  to 
destroy  any  tanks  within  their  firing  range.  Members'  tanks  were 
armed  and  could  fire  at  pillboxes,  but  they  had  to  replenish  supplies 
at  one  of  twelve  refueling  bases  when  ammunition  was  depleted. 
Teamwork  is  required  in  Bolo,  because  while  it  is  extremely  difficult 
for  a  single  task  to  destroy  a  pillbox,  tanks  working  together  can 
readily  do  so. 

8 

Naval  Combat  Experience 

Dangerous  Waters 

Somewhat 

Relevant 

The  Multiplayer  multi-station  mode  allows  players  to  man  a  specific 
station  aboard  a  ship,  plane  or  submarine,  with  other  players  taking 
the  role  of  other  crewmembers  on  the  same  platform.  As  the 
commander  of  the  platform  the  player  can  either  relinquish  control  of 
various  stations  to  the  automated  Autocrew  or  man  all  stations 
automatically.  Upon  selecting  a  platform  and  the  mission  difficulty 
level,  the  player  will  be  provided  with  an  entirely  random  and  dynamic 
scenario  that  will  be  composed  of  an  infinite  combination  of  mission 
goals,  enemy  forces,  and  random  locations. 
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Scenario 

9 

Helicopter  flight  simulator 

Longbow  2 

Somewhat 

Relevant 

The  three  team  members  worked  together  as  a  pilot,  gunner  and 
radar  specialist  to  operate  the  Apache  helicopter  and  were  charged 
with  conducting  attack  missions  in  challenging  battlefields.  The  goal 
of  each  mission  was  to  fly  into  enemy  territory,  destroy  enemy 
targets,  and  return  safely  to  friendly  territory.  To  accomplish  the 
mission  teams  had  to  navigate  a  fixed  course  of  waypoints,  identify 
and  destroy  all  enemy  targets  encountered,  and,  at  the  same  time 
evade  enemy  attacks  on  their  helicopter.  Missions  concluded  in  three 
different  ways:  a)  when  a  team  reached  the  last  waypoint,  b)  when  a 
team  was  destroyed  by  enemy  fire,  or  c)  when  the  12  minute  time 
limit  expired. 

10 

Radar  monitoring  task 

Team  Event-Based 
Adaptive  Multilevel 
Simulation 
(TEAMSim) 

Somewhat 

Relevant 

TEAMSim  is  a  PC-based  simulation  of  a  radar-tracking  task.  Three- 
person  teams  were  seated  at  simulated  radar  consoles  where 
contacts  with  different  priorities  and  patterns  of  movement  appeared. 
Team  members  communicated  freely  with  one  another  on  a  closed 
communication  system.  Participants  needed  to  learn  how  to  'hook' 
contacts  on  the  radar  screen,  collect  information  to  classify  their 
characteristics,  and  render  an  overall  decision  (take  action  or  clear) 
for  each  contact.  Each  team  member  was  primarily  responsible  for 
one  of  the  three  sectors  designated  on  the  display,  but  each  had 
discretion  to  monitor  and  work  in  their  teammates'  sectors. 
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Scenario 

11 

Reconnaissance 

The  Archimedes 
Combat  Modeling 
Platform 

Somewhat 

Relevant 

The  scenario  depicted  is  a  reconnaissance  scenario.  The  (very 
simple)  playbox  consists  of  a  10x10  grid,  each  element  on  the  grid 
represents  a  distinct  terrain  element.  Each  terrain  element  is 
characterized  by  three  quantities:  trafficability,  which  affects 
movement;  cover,  which  affects  combat  adjudication;  and 
concealment,  which  affects  detection.  Blue  represents  water,  yellow 
open  terrain,  green  forest,  grey  mountain  and  dark  grey  mountainous 
forests.  In  this  scenario,  a  recon  team  (blue  Agents)  traverses  a 
series  of  recon  checkpoints  (black  Agents),  searching  for  an  objective 
(turquoise  Agent)  that  is  under  guard  by  the  enemy  (red  Agents). 

The  blue  Agents'  goal  is  to  locate  the  objective  without  being 
detected  by  the  red  Agents.  The  blue  Agents  are  endowed  with 
behaviors  designed  by  the  analyst  to  enable  them  to  achieve  their 
goal.  These  behaviors  are  dependent  upon  an  Agent  Variable, 
discipline. 

12 

Hostage  Extraction 

Team  Performance 
Assessment 
Technology  (TPAT) 

Highly  Relevant 

The  basic  scenario  entails  a  hostage  extraction  situation  in  an 
invented  country.  A  command  structure  exists,  including 
subordinates,  superiors,  and  lateral  colleagues,  composed  of  a  total 
of  nine  personnel,  each  of  which  occupies  a  different  role  (TPAT 
simulates  missing  personnel  if  needed).  Generally,  three  teams  are 
formed  (though  the  option  to  reorganize  them  exists):  (1)  command 
team,  (2)  air  resources  team,  and  (3)  ground  resources  team.  All 
personnel  can  communicate  with  one  another.  Most  information 
(generally  presented  in  the  form  of  a  message)  given  to  a  participant 
is  specific  to  that  participant’s  role.  Based  on  such  information, 
decisions  are  made  -  this  is  accomplished  by  choosing  from  over 

1,000  decision  alternatives.  For  each  decision  made  (which  may  be  in 
response  to  information,  in  response  to  a  request  made  by  another 
participant,  or  to  self-  initiate  an  action),  four  steps  are  taken:  (1)  the 
decision  is  made,  (2)  the  prior  events  leading  to  the  decision  are 
named,  (3)  plans,  if  any,  are  stated,  and  (4)  other  participants  are 
notified  of  the  decision.  TPAT  also  generates  challenges  (e.g., 
emergencies,  sudden  changes,  missing  information)  which  may  alter 
the  course  of  a  decision. 
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13 

Anti-submarine  Warfare 

Hierarchical  Task 
Analysis  (Teams)  - 
HTA  (T) 

Relevant 

The  scenario  entails  four  ships  (provided  to  operators)  which  are 
used  to  escort  and  prevent  the  attack  of  two  highly-  valued  units  (e.g., 
supply  ships).  Anti-submarine  warfare  teams  are  distributed  with  a 
central  control  unit.  These  teams  must  respond  to  specific  events 
(e.g.,  a  request,  a  torpedo  report,  failing  sonar)  by  identifying  threats 
(e.g.,  determining  if  a  presence  is  a  submarine  or  a  whale),  deciding 
threat  location,  and  attacking  if  necessary.  Each  event  is  designed  to 
elicit  specific  actions  (typically  information-sharing  behaviors). 

14 

Dyads  (a  pilot  and  co-pilot) 

Low-Fidelity  Aviation 
Research 

Methodology 

Relevant 

This  paradigm  was  designed  to  study  dyads  (a  pilot  and  co-pilot)  and 
is  available  commercially  as  a  standard  aviation  simulation  program. 
The  hardware  arrangement  involves  only  one  computer  and  two 
monitors,  and  is  therefore  a  low-cost,  relatively  hassle-free 
assessment  that  successfully  provides  elements  of  task  coordination, 
communication,  interdependency  of  roles,  and  measurement  of 
individual  characteristics.  However,  it  may  difficult  to  alter  some 
variables  such  as  task  and  work  characteristics.  Additionally,  the 
configuration  cannot  be  adjusted  and  so  team  size  cannot  be  varied. 
Reliability  and  validity  have  been  found  to  be  acceptable. 

15 

C3  environment  -  monitoring  & 
prosecuting  of  incoming  targets  on 
a  radar  screen 

Team  Performance 
Assessment  Battery 
(TPAB) 

Somewhat 

Relevant 

TPAB  uses  “synthetic  work”  -  in  this  case  the  monitoring  and 
prosecuting  of  incoming  targets  on  a  radar  screen  -  to  approximate  a 
command,  control,  and  communication  environment. 
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16 

Seeking  to  discover  the  intent  of 
targets  based  on  characteristics 

Team  Interactive 
Decision  Exercise  for 
Teams  Incorporating 
Distributed  Expertise 
(TIDE2) 

Highly  Relevant 

Four  team  members,  each  with  a  different  area  of  “expertise,”  seek  to 
discover  the  intent  of  targets  based  on  nine  characteristics.  Five  rules 
exist  that  describe  various  combinations  of  characteristics,  which  then 
dictate  threat  level,  so  participants  must  share  information  in  order  to 
glean  threat  level  to  take  appropriate  action.  Participants  must 
complete  individual  as  well  as  team  tasks.  Similar  to  occupations 
found  in  real  life,  TIDE^  contains  a  hierarchical  organization  -  there  is 
a  designated  leader.  And,  as  in  real  life,  this  leader  can  receive  input 
from  subordinates,  but  may  make  a  decision  at  any  time  -  with  or 
without  the  agreement  or  contribution  of  others.  One  advantage  that 
is  generated  by  this  capability  is  that  group  conflict  can  be 
manipulated.  Other  task  characteristics  such  as  ambiguity,  time 
pressure,  and  task  complexity  can  also  be  modified,  and  this 
paradigm  has  been  used  to  study  the  relationships  between  stress, 
uncertainty,  conflict,  coordination,  and  performance.  TIDE^  scenarios 
can  be  tailored  to  a  variety  of  environments  (e.g.,  naval,  investment 
banking,  personnel  selection),  and  components  for  data  collection, 
data  sorting,  and  data  analysis  are  included. 

17 

A  metropolis  crisis  control  center 

C3  Interactive  Task 
for  Identifying 

Emerging  Situations 
(CITIES) 

Highly  Relevant 

CITIES  is  actually  comprised  of  an  entire  simulation  system  -  not  just 
software  -  in  an  attempt  to  more  closely  approximate  the  constraints 
of  the  environment  that  is  simulated:  a  metropolis  crisis  control 
center.  Team  decision-making  under  pressure  is  the  primary  process 
of  interest;  information  is  distributed  among  participants  and  may  be 
ambiguous,  and  the  task  is  timed.  Operating  the  “crisis  center” 
requires  constant  monitoring  and  assessment  (situation  awareness) 
of  changing  events,  as  well  as  shifting  the  allocation  of  resources  as 
priorities  change.  Teams  are  geographically  dispersed  and  composed 
of  two  members,  but  CITIES  is  unique  in  that  two  cooperative  two- 
member  teams  (fire  and  police)  work  together,  thereby  introducing  a 
communication  pattern  not  often  examined  in  most  team  process 
assessments.  Communication  between  team  members  is  by  audio 
conferencing  and  two-way  video,  supplemented  by  monitors  with 
interactive  touch  screens.  The  type,  number,  sequence,  timing, 
location,  and  severity  of  events  can  be  manipulated,  as  can 
information  richness  and  resource  availability.  Most  dependent 
measures  are  recorded  electronically  by  the  system. 
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18 

A  radar-like  target  classification  task 

Team  Argus 

Somewhat 

Relevant 

Team  Argus  is  solely  a  decision  task.  In  the  decision  task,  information 
is  displayed  to  the  subject  about  possible  targets  in  the  air  space  and 
on  the  ground.  Based  on  this  information  and  criteria  given  to  the 
subject,  he  or  she  is  to  make  a  decision  about  what  action  to  take 
(e.g.,  monitor,  warn,  engage  etc.).  In  Team  Argus  access  to  the  exact 
values  of  target  attributes  (cues)  is  distributed  among  the  members. 
Members  can  send  their  cue  data  to  other  team  members  and  can 
request  data  from  other  members.  In  team  Argus,  communication 
between  team  members  is  accomplished  through  text  and  data 
messages  sent  between  workstations. 

19 

Recovering  weapons  from  hidden 
caches 

Neverwinter  Night 

Relevant 

The  game-based  testbed  uses  a  “re-skinned”  version  of  the 
commercial  off-the-shelf  medieval  fantasy  role-playing  game 
Neverwinter  Nights.  The  re-skinned  version  features  a  simulated 
modern  cityscape  with  people  wearing  modern  clothes  and  engaged 
in  non-magical  activities.  Basically,  team  members  are  assigned  roles 
(e.g.,  patrol  leader,  weapons  specialist)  depending  on  the 
experimental  condition  and  the  team  is  given  the  high  level  task  of 
locating  and  acquiring  caches  of  weapons  hidden  within  a  town.  The 
team  is  provided  with  equipment  to  help  with  the  task  (sensors  of 
varying  capabilities  designed  to  help  locate  weapons  caches  and 
tools  for  opening  doors  and  crates)  and  must  decide  how  to  allocate 
those  resources.  Additionally,  team  members  have  collaboration 
tools,  allowing  information  to  be  shared  between  individuals  and 
locations  flagged  or  marked  within  the  virtual  environment. 

20 

North  African  “insertion  from  the 
sea” 

DDD-III  simulator 

Highly  Relevant 

The  primary  mission  tasks  include  demining  and  taking  two  beaches, 
taking  and  holding  a  hill  overlooking  one  of  the  beaches,  identifying 
and  destroying  the  correct  bridge  to  prevent  enemy  forces  from 
reinforcing  enemy  troops  near  the  beach  heads,  taking  and  holding 
an  airfield,  and  taking  and  holding  a  sea  port.  In  addition  to  the  main 
mission  tasks  the  team  must  contend  with  a  number  of  additional 
tasks  that  arise  at  unexpected  times  during  the  scenario  trial  (e.g., 
suppressing  enemy  artillery,  destroying  FROG  missile  launchers, 
destroying  enemy  aircraft,  destroying  enemy  armor). 
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21 

Capture-the-flag  event  at  a  “Platoon 
vs.  Platoon”  level. 

Neverwinter  Nights 

Relevant 

The  exercise  scenario  was  similar  to  a  capture-the-flag  event  at  a 
“Platoon  vs.  Platoon”  level.  The  participants  were  separated  into  two 
competing  platoons  of  size  20.  Each  platoon  was  comprised  of  3 
squads,  each  with  a  similar  mix  of  player  types.  Each  participant  was 
assigned  a  specific  avatar  and  a  role  (platoon  leader,  squad  leader  or 
squad  member).  The  avatars  varied  in  their  individual  capabilities  in 
order  to  promote  teamwork  and  collaboration  between  players  as 
they  had  to  cooperate  to  apply  different  combinations  of  capabilities 
in  order  to  meet  mission  demands.  In  each  of  two  camps,  a  flag  was 
placed  which  indicated  possession  or  ownership  of  that  territory. 
Adjacent  to  the  flag  was  a  lever  that  was  used  to  indicate  a  change  in 
possession  of  that  flag  and  surrounding  territory.  Once  a  lever  was 
pulled,  it  would  remain  in  the  possession  of  the  puller’s  platoon  until  a 
member  of  the  opposing  platoon  gained  access  to  the  lever,  thereby 
claiming  it  as  their  own.  In  addition  to  the  levers  in  the  two  camps,  a 
third  flag  and  lever  were  located  in  a  “Hidden  Camp.”  Players  were 
told  that  the  hidden  lever  could  be  found  somewhere  in  the 
environment,  but  were  not  given  its  specific  location.  The  hidden 
camp  was  protected  by  NPCs  who  could  inflict  damage  to  the 
avatars.  The  stated  goals  of  the  game  were  to  defend  your  flag  while 
capturing  the  flag  of  the  opposing  team  or  of  the  hidden  camp. 

22 

Radar-based  ATC  Tasks 

ATC  team  training 
device 

Somewhat 

Relevant 

Three  scenarios  -  low,  medium,  and  high-density  conditions  -  were 
developed  and  calibrated  to  create  three  levels  of  aircraft  density 
based  on  the  number  of  aircraft  presented  to  the  team  over  time. 
Aircraft  originally  appeared  in  an  inactive  state  and  were  activated  at 
the  discretion  of  the  participant  in  the  originating  sector.  Once 
activated,  the  aircraft  had  to  travel  through  three  sectors  before 
landing  at  an  airport  in  the  fourth  sector.  The  tasks  required  that 
subjects  use  a  point  and  click  method  with  a  mouse  to  issue  changes 
in  aircraft  direction,  speed,  and  altitude. 
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23 

Ensuring  that  launching  a  space 
vehicle  will  not  endanger  the 
general  populace 

Multi-agent  Operation 
Range  Simulation 
Environment 
(MORSE) 

Highly  Relevant 

It  is  a  distributed  system  that  simulates  a  task  performed  by  a  team  of 
three  human  operators,  each  being  responsible  for  some  aspect  of 
ensuring  that  launching  a  space  vehicle  from  KSC  will  not  endanger 
the  general  populace.  MORSE  provides  monitoring  stations  where 
human  operators  track  the  progress  of  the  mission  prior  to  launch, 
and  decide  whether  to  abort  or  go  ahead  with  the  launch.  To  make 
this  decision,  the  team  must  monitor  incursions  (e.g.,  aircraft  or 
boats)  that  have  entered  the  exclusion  zone,  an  area  bounded  by 
launch  impact  lines.  The  human  team  has  at  its  disposal  radars  that 
allow  the  team  members  to  see  incursions  in  the  areas  covered  by 
the  radars,  and  interceptor  vehicles  that  can  be  appointed  to  intercept 
incursions.  Radars  and  interceptor  vehicles  are  shared  resources  that 
team  members  must  acquire  through  coordination  with  each  other 
and  utilize  for  the  performance  of  the  team  task.  The  overall  team 
objective  is  to  effect  a  safe  launch  where  no  incursions  are  left  within 
the  impact  lines  at  launch  time  and  resource  consumption  over  the 
course  of  the  task  is  minimized. 

24 

Working  together  to  find  Scud 
launchers 

SCUDHunt 

Highly  Relevant 

SCUDHunt  is  an  abstract  game  of  command  and  control, 
implemented  as  a  multi-player  web-based  game.  Team  members 
play  via  the  Internet;  they  may  be  collocated  or  distributed.  The  goal 
of  the  game  is  for  team  members  to  correctly  determine  (within  a 
specified  number  of  turns)  where  three  Scud  launchers  are  located 
on  a  5-by-5  grid.  The  three  launchers  are  randomly  placed  at  the  start 
of  each  game  and  remain  stationary.  Once  the  players  find  them, 
they  do  not  move.  Each  player  is  assigned  a  role  as  an  asset 
manager  and  controls  one  or  more  sensor  assets.  Each  sensor 
covers  a  different  number  of  squares  and  has  a  different  probability  of 
detecting  a  launcher.  Some  assets  can  also  return  false  information, 
requiring  verification  on  a  subsequent  turn  by  a  more  reliable  asset. 
Conceptually,  each  turn  has  three  phases:  1)  a  search  phase,  in 
which  players  place  their  assets  on  the  board  and  receive  search 
results,  2)  a  phase  in  which  players  share  results,  and  3)  a  strike  plan 
phase,  in  which  players  nominate  those  squares  they  feel  most  likely 
contain  a  SCUD  launcher. 
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25 

Naval  Air  Defense  Warfare 

Wright  State  Aegis 
Simulation  Platform 
(WASPyieam  Aegis 
Simulation  Platform 
(TASP)  (an  extension 
of  WASP) 

Somewhat 

Relevant 

WASP  uses  a  simplified  naval  command  and  control  team  consisting 
of  a  Tactical  Action  Officer  (TAO),  an  Identification  Supervisor  (IDS), 
and  the  Electronic  Warfare  Supervisor  (EWS).  We  assume  that  each 
member  of  the  team  must  accomplish  specified  tasks  based  on  the 
tactical  situation,  rules  of  engagement,  and  commanding  officer  (CO) 
directives.  The  TAO  supports  the  CO  to  command,  maneuver  the 
ship,  and  define  operational  parameters.  The  TAO  also  can  give  the 
order  to  deploy  weapons  and  provide  verbal  feedback  to  team 
member  requests.  The  IDS  is  responsible  for  the  supervision  and 
control  of  the  identification  function.  The  IDS  controls  the 

Identification  Friend-Foe  (IFF)  functions,  manages  air  tracks  in  the 
system,  and  issues  verbal  level  warnings  to  perceived  hostile  aircraft. 
The  EWS  is  responsible  for  supervising  the  operation  of  electronic 
support  measures.  Furthermore,  the  EWS  is  responsible  for  the 
proper  characterization  of  tracks  and  association  of  sensors  to  tracks. 

26 

Janus  wargame 

Wargaming 

simulation 

Relevant 

Janus  is  an  interactive  simulation  wargame  that  allows  multi-sided 
combat,  under  realistic  simulation  conditions.  The  view  on  the  screen 
is  a  two-dimensional  map  with  grid  lines.  However,  to  the  weapon 
systems  the  terrain  is  three-dimensional  (with  elevation  and  contour 
lines).  This  affects  maneuverability  and  line  of  sight.  Unless  the 
enemy  is  in  the  line  of  sight,  each  player  can  only  see  their  own 
forces.  When  an  enemy  enters  the  line  of  sight  of  a  friendly  unit,  it  is 
spotted  and  identified.  The  units  then  automatically  engage  with  the 
enemy,  making  decisions  about  what  type  of  ammunition  they  will  use 
(this  decision  is  programmed  into  the  computer’s  database).  Janus 
models  real  world  phenomena,  including  natural  and  man  made 
obstacles,  the  requirement  for  route  planning,  direct  and  indirect  fire 
engagements,  planned  fire  missions,  and  tactical  decision-making  as 
the  game  progresses.  The  tasks  require  the  coordination  and  co¬ 
operation  of  a  team  of  people.  Janus  allows  a  team  of  three  or  more 
participants.  One  participant  can  act  as  a  battle  commander,  while 
others  can  act  as  sub-unit  leaders.  This  sets  up  a  requirement  for 
distributed  decision-making.  A  communication  system  can  also 
be  used  between  the  team  members. 
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27 

Fire  Fighting 

Networked  Fire  Chief 
(NFC) 

Relevant 

It  requires  the  operator  to  make  decisions  under  continually  changing 
conditions.  Participants  are  required  to  fight  fires  that  spontaneously 
break  out  on  a  map,  thus  providing  an  element  of  uncertainty.  Also, 
the  tasks  involved  in  successfully  fighting  a  fire  require  the  co¬ 
ordination  and  co-operation  of  a  team  of  people.  An  advantage  of  the 
NFC  program  is  that  it  can  be  networked,  so  that  hierarchical 
organization  structures  can  be  examined.  Variations  in  weather  are 
represented  by  wind  direction  and  wind  speed.  The  participants  are 
required  to  use  the  appliances  to  extinguish  the  fires.  The  appliances 
have  some  of  the  same  limitations  as  their  real  world  equivalents. 

The  decision-maker  is  also  required  to  prioritize  areas  when 
allocating  resources  to  fight  the  fires.  The  order  of  priority  established 
on  NFC  is:  1)  residences,  2)  pastures,  3)  national  park,  and  4) 
grassland.  The  purpose  of  this  variety  in  landscape  is  to  create  a 
more  complex  and  realistic  goal.  Performance  scores  show  the 
percentage  of  landscape  left  after  a  designated  period  of  time  had 
elapsed,  taking  land  priority  into  consideration. 
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28 

Fire  fighting 

CSFire 

Highly  Relevant 

The  environmental  domain,  which  is  forest  fire  fighting,  is  of 
subsidiary  interest  and  has  been  chosen  because  it  generates  a  good 
dynamic  target  system.  The  system  generates  a  task  environment  in 
which  a  group  of  people  cooperate  to  extinguish  a  forest  fire.  The 
simulation  includes  forest  fire(s),  different  kinds  of  vegetation, 
infrastructure  (“villages”  and  “cities”  -  that  are  represented  as 
houses),  computer-simulated  agents  (fire-fighting  units  and 
reconnaissance  personnel).  The  user  interface  consists  of  three  basic 
elements;  (1)  a  Geographic  Information  System  (GIS);  (2)  a  diary; 
and  (3)  an  e-mail  system.  The  GIS  can  be  manually  or  automatically 
updated,  as  well  as  shared  with  other  users.  During  a  session,  the 
simulator  updates  the  GIS  around  the  simulated  units  for  the  actors 
who  are  controlling  these  units.  The  players  who  run  the  system  are 
part  of  a  firefighting  organization  and  can  take  on  the  roles  of 
decision-team  members  or  fire-fighting  unit  chiefs.  The  task  of  the 
decision-team  is  to  have  an  overview  of  the  situation,  and  to  co¬ 
ordinate  and  schedule  the  fire-fighting  units  so  that  they  can 
extinguish  the  fire  and  save  the  infrastructure.  Communication  among 
the  different  organizational  parts  is  mainly  conducted  through  mail 
and  GIS  updates 

29 

Joint  command  and  control 

DDD-III 

Highly  Relevant 

Basically,  a  country  friendly  to  the  United  States  has  been  invaded  by 
a  neighboring  state  and  has  asked  the  United  States  for  help.  In 
response,  a  Joint  Task  Force  (JTF)  is  tasked  to  conduct 
expeditionary  amphibious  operations  to  seize  a  seaport,  an  airport 
and  a  key  bridge  to  facilitate  the  introduction  of  follow-on  forces.  The 
forces  must  accomplish  a  set  of  approximately  50  tasks,  some  known 
and  some  surprise,  and  some  with  temporal  interdependencies,  to 
achieve  the  overall  mission.  Developing  concepts  and  methods  to 
design  architectures  optimally  matched  to  such  a  set  of  tasks,  and 
comparing  the  performance  of  these  architectures  against  that  of 
more  traditional  architectures  have  been  key  foci  of  the  A2C2 
research.  “Trigger”  events  that  dramatically  alter  the  task  set  or 
resources  available  have  also  been  introduced  during  the  scenario  in 
two  of  the  experiments  (two  and  three)  to  examine  structural  and 
process  adaptation. 
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Scenario  name 

Platform  Name 

Relevance 

Scenario 

30 

Airborne  Warning  and  Controi 

System  (AWACS)  Weapons 

Director  Teams 

DDDnet 

Highiy  Reievant 

This  version  of  the  DDDnet  was  deveioped  to  represent  the 
underiying  cognitive  and  decision  making  task  demands  of  Airborne 
Warning  and  Controi  System  (AWACS)  Weapons  Director  Teams, 
based  on  muitipie  investigations  of  cognitive  and  functionai  aspects  of 
this  performance  domain  (Coovert,  et  ai.,  2001).  Further  deveiopment 
resuited  in  a  scenario  that  emuiates  three  miiitary  C2  teams:  the 

USAF  AWACS  team,  another  USAF  ground-based  C2  team,  and  a 
third  Navy  airborne  C2  team.  The  AWACS  DDD-Net  was 
impiemented  and  demonstrated,  aiiowing  distributed  simuiations  over 
the  Internet.  It  linked  the  different  locations,  allowed  multi-role 
missions,  data  collection,  and  feedback.  Different  parts  of  the 
network  included  1-2  connections  for  improved  speed  and 
performance.  The  DDDnet  achieved  and  maintained  a  synchronized 
connection  for  an  AWACS  simulation  involving  16  participants. 
Simultaneously,  observers  at  each  location  rated  performance  using 
web-based  tools  that  allowed  immediate  data  pooling,  analysis,  and 
feedback,  within  10  minutes  after  data  input  was  complete. 

31 

Airborne  Warning  and  Controi 

System 

C3STARS 

Somewhat 

Reievant 

The  crew  stations  and  scenarios  simulate  the  air  defense  mission  of 
an  AWACS  (Airborne  Warning  and  Control  System)  platform. 

Realism  is  achieved  through  the  functional  representation  of 
equipment  and  displays,  experienced  personnel  playing  the  role  of 
simulation  pilots,  and  the  use  of  operational  scenarios. 
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Scenario 

32 

Joint  Task  Force  (JTF) 

DDD-III 

Highly  Relevant 

The  U.S.  is  taking  action  in  support  of  an  ally,  Country  Green,  that 
has  been  invaded  by  neighboring  Country  Orange.  The  ultimate 
objectives  of  this  mission  are  to  secure  Country  Green’s  Airport  and 
Port.  A  mission  briefing  document  that  outlined  a  specific  chronology 
of  mission  tasks  to  be  undertaken  by  the  JTF  was  distributed  to  all 
participants.  A  greatly  simplified  version  is  listed  below.  1. 

Amphibious  forces  will  land  and  take  North  and  South  Beach  after 
clearing  mines.  2.  Prior  to  taking  N.  Beach,  infantry  (INF)  and  air 
support  will  seize  and  hold  the  overlooking  the  beach.  3.  Infantry  will 
move  down  roads  from  S.  Beach  toward  airport  and  from  N.  Beach  to 
Port  clearing  mines  and  enemy  tanks.  4.  Special  Operations  Force 
(SOF)  and  satellite  (SAT)  must  determine  which  of  two  roads  the 
enemy  plans  to  use  for  insertion  of  forces  by  assessing  traffic.  Once 
the  enemy  “lead  vehicle”  is  identified,  it  should  be  destroyed  as  well 
as  the  bridge  being  used  by  that  vehicle,  while  retaining  second 
bridge  for  friendly  traffic.  5.  Armored  counterattack  forces  are 
believed  to  be  at  the  Airport  and  Port.  If  present,  the  must  be 
identified  and  destroyed.  6.  Both  the  Port  and  Airport  must  be 
captured  and  held.  The  attack  on  the  Airport  has  priority  and  should 
occur  first  if  they  cannot  be  attacked  simultaneously. 
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Scenario 

33 

Pilots  (ATC) 

Microsoft  Flight 
Simulator 

Highly  Relevant 

Two  scenarios  were  developed  following  National  Aviation  and  Space 
Administration  (NASA)  and  Naval  Training  Systems  Center  (NTSC) 
guidelines  for  air  crew  coordination  skills  scenario  design.  Pilots 
began  each  scenario  on  the  runway  of  an  airport  in  Florida.  In  both 
cases,  they  were  told  that  their  mission  was  to  fly  an  admiral  from  one 
airport  to  another.  In  both  cases  various  things  went  wrong  during  the 
flight  which  required  the  teams  to  cope  in  various  ways.  In  Scenario 

A,  the  pilots  flew  from  Orlando  to  Daytona  Beach.  The  airport  at 
Daytona  was  closed,  so  that  the  pilots  had  to  land  at  their  alternative 
landing  site,  Ormand  Beach.  Pilots  also  lost  communication  with  the 
air  traffic  controller  temporarily.  In  Scenario  B  the  pilots  flew  from  St. 
Augustine  to  Malcolm  McKinnon  in  Georgia.  Near  the  state  line,  the 
admiral  had  a  heart  attack,  and  the  pilots  had  to  fly  to  the  nearest 
emergency  medical  facility,  Jacksonville  International,  to  land.  As 
they  were  descending,  the  runway  was  closed,  forcing  them  to  land 
on  another  runway  at  Jacksonville.  An  actual  air  traffic  controller 
participated  during  the  scenarios.  He  also  played  the  other  two  roles 
needed  during  the  flights:  that  of  Chief  Jones,  who  accompanied  the 
admiral  during  the  flight,  and  the  contact  person  for  Command  Net,  to 
whom  the  pilots  reported  during  flight. 
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Scenario 

34 

Joint  Task  Force  Group  (JTFG) 

DDD  III 

Somewhat 

Relevant 

In  a  bellicose  interstellar  realm  of  the  Third  Millennium,  a  Joint  Task 
Force  Group  (JTFG)  is  given  a  set  of  resources  and  is  assigned  to 
conduct  a  multi-faceted  operation  aimed  at  attacking  and  destroying 
three  strategic  enemy  objects:  a  Geothermal  Power-Plant,  a  Radar 
Tower,  and  a  Sonar  Station.  From  intelligence  sources,  it  is  known 
that  the  enemy  units  used  to  protect  the  above  objects  include  missile 
towers  (named  Pulverisers),  stationary  plasma  batteries  (Punishers), 
and  Light  Laser  batteries.  It  is  also  known  that,  due  to  a  significant 
energy  consumption  while  in  their  active  mode,  the  Punisher  and 

Light  Laser  battery  units  (the  exact  number  of  which  is  unknown)  are 
not  activated  until  they  receive  an  alarm  message  from  appropriate 
enemy  intelligence  units.  Until  their  activation,  the  Punisher  and  Light 
Laser  battery  units  are  invisible  to  energy  sensors  (which  is  another 
reason  for  keeping  them  inactive  until  the  battle  commences).  It  is 
estimated  that  it  takes  the  enemy  one  minute  to  activate  each 
additional  unit.  On  the  other  hand,  the  Pulverisers  remain  in  constant 
readiness  and  can  fire  at  any  given  time.  In  addition  to  the  above 
information,  friendly  intelligence  reports  that  the  enemy  is  using  five 
hovercraft  air  scouts  (termed  Peepers)  for  land  recognizance.  While 
Peepers'  main  function  is  to  spot  any  land  warfare  advancing  toward 
the  Power-Plant,  Radar  Tower,  and  Sonar  Station,  and  to  send  the 
alarm  message  to  all  enemy  units  tasked  to  protect  these  objects. 
Peepers  can  also  carry  various  air-to-ground  missiles  and  can  be 
used  to  destroy  the  land  warfare  units.  The  composition  of  friendly 
assets  (with  versatile  capabilities)  available  for  the  operation  is  as 
follows:  (i)  an  air-fighter  (Avenger);  (ii)  a  stealth  airfighter  (Hawk);  (iii) 
a  fast  attack  vehicle  (Jeffy);  (iv)  a  mobile  rocket  launcher  (Merl);  (v)  a 
very  heavy  assault  tank  (named  Goliath);  (vi)  an  amphibious  tank 
(Triton);  (vii)  a  mobile  radar  (Informer);  and  (viii)  a  heavy  assault  tank 
(Reaper).  The  friendly  forces  are  initially  located  in  the  middle  bottom 
portion  of  the  map.  The  enemy  objects  are  located  in  the  upper 
section  of  the  map  in  a  dormant  volcanic  area,  and  Peeper  units  are 
scattered  across  the  map,  hovering  over  the  no-flight  zone  that 
separates  the  friendly  forces  from  the  enemy  objects. 
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Scenario 

35 

Team  C2  Task  under  sustained 
operation  environment 

AEDGE™  (Agent 
Enabled  Decision 
Guide  Environment) 

Highly  Relevant 

Atypical  scenario  include  a  pre-mission  planning  section,  the 
scenario  itself,  and  a  post-mission  debrief  section.  Within  scenarios, 
three  different  roles  were  utilized,  consisting  of  ISR  (Intelligence, 
Surveillance,  and  Reconnaissance,  Strike,  and  Sweep.  They 
participate  as  a  three  person  teams.  They  own  assets  related  to  their 
functions  respectively.  Scenarios  were  constructed  to  require  high 
amounts  of  interdependence  coordination  among  the  functions. 

36 

International  humanitarian 
assistance 

N/A 

Somewhat 

Relevant 

A  situation  has  arisen  in  a  Central  African  country  that  has  placed  a 
large  number  of  lives  at  risk.  The  UN  has  asked  for  a  Canadian 
contribution.  The  Government  has  agreed.  Parliament  has  been 
consulted  and  an  appropriate  Order-in-Council  has  been  enacted. 

The  CF  contingent  mission  is  to  transport  UN  humanitarian  supplies 
from  a  UN  forward  staging  area  to  local  (NGO)  staging  areas  for 
further  dissemination  by  NGOs  and  local  relief  agencies.  The  CF  will 
also  command  one  of  the  UN  forward  staging  areas  and  supply  the 
NGOs  with  support  services  at  the  staging  area.  CF  faces  uncertainty 
as  to  unexpected  situations  and  tasks. 

37 

Peace  support  operations 

N/A 

Somewhat 

Relevant 

Tension  between  two  non-NATO  bordering  states  has  escalated  to 
include  armed  conflict.  One  state  is  likely  to  attain  an  overwhelming 
victory  over  its  opponent.  UN  Security  Council  passed  a  resolution 
that  a  multinational  force  under  UN  command  will  be  formed  and 
deployed  to  restore  the  previous  situation.  Canada  has  agreed  to 
deploy  maritime,  land  and  airforce  personnel  abroad.  CF  Mission  is 
to  achieve,  as  part  of  a  coalition  operation,  the  withdrawal  of  Swasian 
forces  from  Weston  territory  through  the  contribution  of  land,  sea  and 
airforces. 
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Annex  F:  Comparison  of  Computational  Models 


(Note:  Model  labelled  with  represents  email  questionnaire  responses  are  available  for  it  and  therefore  the  judgements  for  the  model  is  mostly  based  on  the 
questionnaire  responses.) 


No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

1 

IPME* 

(Integrated 

Performance 

Modelling 

Environment) 

Micro  Analysis  &Design  (MA&D) 

http :  //WWW .  maad.  com 

(full-featured  discrete-event  simulation 
environment) 

Yes 

N/A 

Yes 

Yes 

Yes 

Yes 
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Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

2 

IMPRINT* 

(Improved 

Performance 

Research 

INtegration 

Tool) 

Micro  Analysis  &Design  (MA&D)  for  Army 
Research  Laboratory  (ARL) 

http :  //WWW .  arl .  army .  mil/  ARL- 
Directorates/HRED/imb/imprint/Imprint7.htm 

(event-based  task  network) 

Yes 

(evaluate  operator  &  crew 
workload) 

-  It  does  not 
automatically 
import/export 
data  to  IPME. 
However, 
IMPRINT 

shares  the  same 
task  network 
modeling 
simulation 
engine 

with  IPME. 

-  IPME  is  part 
of  IMPRINT 

IMPRINT/ACT 
-R  Integration 
(Ref  3,  4,  5) 

CART 

(Combat 

Automation 

Requirements 

Testbed), 

another  example 

of 

Computational 

Task 

Simulation,  is 
built  on 
IMPRINT. 

Yes 

Yes 

Yes 

Yes 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

3 

ACT-R/PM 

(Adaptive 
Control  or 
Atomic 

Components  of 
Thought  - 
Rational/Perce 
ptual-Motor) 

Carnegie  Mellon  University  (CMU) 

http :  //act-r .  ps  v .  emu .  edu/ 

(a  "hybrid"  cognitive  architecture  that  aspires 
to  provide  an  integrated  account  of  many 
aspects  of  human  cognition) 

No 

No 

Yes 

Yes 

No 

No 

4 

MIDAS* 

(Man-machine 
Integration 
Design  and 
Analysis 
System) 

NASA  Ames  Research  Center  San  Jose  State 
University  and  QSS  Group,  Inc. 

http :  /  /human- factors .  arc .  nasa.  gov/ dev/www- 

midas/ 

(Generalized  its  predictive  capability  to  areas 
of  design  outside  of  the  original  application  - 
helicopter  application  domain.  For  example 
operations  in  commercial  aviation,  space 
shuttle  design.  Crew  Exploration  Vehicle 
design  and  even  to  scope  experimental  timing 
for  glovebox  experiments  conducted  in 
microgravity  conditions  aboard  the 
International  Space  Station. 

Yes 

(Measures  it  using  Wicken's 
multiple  resource  theory.) 

Will  be 

(Originally  is  a 
standalone 
constructive 
simulation 
system  that  has 
not  been  used  to 
interact  with 
other  models  or 
military 
simulations. 

Now  is 
collaborating 
with  MA&D  to 
develop  a  new 
version  that 
probably  will  be 
compatible  with 
IPME. 

Yes 

Yes 

(geared 
towards 
human 
operators 
interacting 
with  and 
within  a 
crewstation  of 
any  kind, 
which  may  be 
contained  in  a 
vehicle, 
moving  within 
an  outdoor 
environment) 

No 

Yes 

(“Multi- 
Agent”  , 
capable  to 
model 
multiple 
crewmembers 
and 

crewstations) 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

5 

D-OMAR* 

(Distributed  - 
Operator 
Model 

Architecture) 

BBN  Technologies  under  sponsorship  from  the 
ARE.  http://omar.bbn.com/ 

(The  multiagent  architecture  of  OMAR  is 
especially  suited  to  model  social  interactions. 

Eor  the  ATC  model,  OMAR  models  in-person 
interactions  among  aircrew  members, 

“party-line”  radio  communications  among 
aircrews,  and  telephone  communication 
between  ATC  centers  in  adjacent  sectors.) 

No 

(Very  detailed  trace  of  model 
goal  and  procedure  execution 
from  which  workload  can  be 
computed  according  to 
requirements.) 

Probably 

(was 

successfully 
linked  with  Air 
Eorce’s  DCOG, 
ACT-R, 
COGNET/iGE 

N  and  EPIC- 
Soar  and  used 
to  control  Jack* 
an  animated, 
interactive 
virtual 

simulation  of  a 
human) 

(also  providing 
synthetic  team 
members  for 
DDD) 

Yes 

(Simply  a  set  of  software  tools 
for  building  real-time  or  fast¬ 
time  simulations  with  no 
constraints  on  the  domain 
simulated  or  the  scenarios  that 
may  be  constructed.) 

(Primarily  to  simulate 
behaviour  related  to  ATC) 

Yes 

(Not  usual 
ways  been 
used.  Could 
readily  be 
done,  if  one  is 
interested  in 
large 

organizational 

modeling 

problems.) 

Yes 

(Way  of 
regularly 
being  used 
with  great 
attention  to 
the 

interactions  of 
individuals 
within  the 
team.) 

6 

SAMPLE 

(Situation 
Awareness 
Model  for 
Pilot-in-the- 
loop 

Evaluation) 

Charles  River  Analytics  (CRA)  Inc. 

http :  //mentalmodels .  mitre .  org/cog  eng/referen 
ce  documents/ sample  %  20model  %  20for  %  20pil 

ot-in-the-loop  %  20evaluation.pdf 

No 

(model  SA) 

Has  not  been 
interfaced  with 
other  cognitive 
models 

(Developed  in 
C-f-f) 

Yes 

Yes 

(expanded  to 
model  SA  in 
other  types  of 
performance 
enviromnent, 
other  than 
original 
purpose:  air 
combat) 

No 

Yes 

(modeling 
individual  and 
crew  SA) 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

7 

GLEAN* 

[GOMS 
(Goals, 
Operators, 
Methods, 
Seleetion 
Rules) 
Language 
Evaluation  and 
Analysis] 

The  Artificial  Intelligence  (AI)  program  at  the 
University  of  Michigan 

http  ://www.eecs.  umich .  edu/  ~  kieras/goms .  ht 

ml 

[A  software  tool  for  constructing  simulation  of 
a  human  user  (programmed  with  GOMS) 
interacting  with  a  simulated  device  or  system.] 
Ref  6 

Yes 

(workload  profiles) 

Probably 

(Recent  version 
of  GLEAN3  is 
grounded  in 
EPIC) 

(Would  need 
work  to 
integrate.) 

Yes 

Yes 

No 

Yes 

(model  human 
performance 
of  a  team  of 
operators) 

8 

APEX* 

(Architecture 
for  Procedure 
Execution) 

NASA  Ames  Research  Center 

http :  //ti .  arc .  nasa .  go  v/pro  i  ects/apex/ 

No 

(No  with  Apex  alone.  Yes  in 
conjunction  with  MIDAS 
which  uses  Apex  as  its 
behaviour  engine.) 

No. 

(However, 
Micro  Analysis 
and  Design  has 
recently 
acquired  a  copy 
of  Apex  (as  part 
of  MIDAS)  and 
is  planning  to 
build  tools  for 
it.  They  may 
have  plans  to 
use  it  with 
IPME  in  some 
way.) 

Yes 

Yes 

(published 
research  has 
been  limited 
to  a  single 
application: 
air  traffic 
control) 

Probably  No 

(At  least  no 
one  seems  to 

have 

attempted  it.) 

Yes 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

9 

COGNET 

(Cognitive  as  a 
Network  of 
Tasks) 

[Integration  of 
COGNET/BA 
TON 

(Blackboard 
Architecture 
for  Task- 
Oriented 
Networks) 
supports 
cooperative 
and 

teamwork 
behaviors.  Ref 
7] 

CHI  Systems 

http :  //WWW .  chisvstems .  com/ 

Yes 

(Metacognitive  functions 
permit  to  model  self- 
awareness  phenomena,  e.g. 
subjective  workload) 

No 

(Written  in 
C/C-f-f) 

Yes 

Yes 

No 

Yes 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

10 

Soar* 

(Historically, 
Soar  stood  for 
State,  Operator 
And  Result) 

[Soar  is  an 
arehitecture 
for  building 
real-time 
cognitive 
systems,  is 
domain 
independent, 
and  has  been 
integrated  with 
various 
simulation 
environments. 

The 

development 
of  a  Soar 
system  is 
generally  done 
by  developing 
agents  by 
writing  rules.] 

Under  active  development  at  several  sites 
around  the  world 

U  of  Michigan: 

http :  / /sitemaker .  umich.edu/soar 

CMU  :http :  //www .  es .  emu.  edu/ afs/ cs/project/ s 
oar/public/w  w  w/home-page .  html 

use :  http :  / /www .  isi .  edu/ soar/ soar- 
homepage.html 

U  of  Nottingham: 

http :  //w  w  w .  nottingham .  ac .  uk/pub/ soar/ 

U  of  Hertfordshire: 

http :  / /phoenix .  herts .  ac .  uk/  ~  rmv/cogarch .  semi 

nar/soar.html 

U  of  Portsmouth: 

http :  //www .  port .  ac .  uk/ departments/ academic/c 

t/ research/soarasanagent/ 

Soar  Technology  Inc: 

http :  // w  w  w .  soartechnology .  com 

Explore  Reasoning  System  Inc.: 
http :  // w  w  w .  ers .  com 

No 

(There's  nothing  in  the 
architecture  to  prevent  this, 
but  it  hasn't  been  done  much 
in  practice.) 

Yes 

(It  has  not  yet 
been  integrated 
with  IPME,  but 
could  be  done 
with  a  bit  of 
work.) 

Yes 

(Depending 
on  how  you 
engineer  your 
Soar  agent 
system.) 

Yes 

(There  is 
nothing  in  the 
architecture 
that  ties  it  to  a 
particular 
domain. 
Although  new 
knowledge 

e-g- 

production 
rules,  will 
need  to  be 
written  for 

new 

domains.) 

Yes 

"Multiple 

Interacting 

Agent" 

(This  has  been 
done  in 
several 
projects.) 

Yes 

(This  has 
been  done  in 
several 
projects.) 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

11 

EPIC 

(Executive 

Proeess- 

Interaetive 

Control) 

The  Brain,  Cognition,  and  Aetion  Laboratory 
at  the  University  of  Miehigan 

http :  /  /  w  w  w  .umich.  edu/  ~  bcalab/ epic .  html 

No 

Many  of  EPIC’s 
features  of  are 
now  embodies 
in  the  GLEAN 
model. 

EPIC  has  been 
successfully 
incorporated 
into  Soar  and 
ACT-R 

Yes 

Yes 

No 

No 

(It’s  a  model 
of  individual 
performance; 

it  has  no 
capability  to 
model 
collective 
behaviour) 

12 

PUMA 

[Performance 
and  Usability 
Modelling  in 
ATM  (Air 
Traffic 

Management)] 

For  National  Air  Traffic  Services  (NATS)  by 

Roke  Manor  Research  Limited, 
http :  / /web .  mit .  edu/aeroastro/ w  w  w/labs/ AATT 

/reviews/puma. html 

Yes 

(It  is  a  toolset  designed  to 
enable  the  prediction  and 
description  of  controller 
workload  for  ATC 
seenarios.) 

Possible 

Yes 

Capable 

(Originally 
developed  for 
air  traffic 
scenario) 

No 

No 

13 

A-SA 

(Computationa 

1  Model  of 
Attention- 
Situation 
Awareness) 

NASA  &U  of  Illinois 

http: //human- 

faetors.arc.nasa.gov/ihi/hcsl/HPM  pubs/Wick 

ens  AvSP  HPM.pdf 

http :  // ww  w  .humanfactors  .uiuc .  edu/Reports&P 

No,  predicts  errors 

No 

(It’s  a 
Conceptual 
Model) 

Capable 

Yes 

No 

No 

apersPDFs/TechReport/04- 1 5 .  pdf 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

14 

Brahms* 

(Business 

Redesign 

Agent-based 

Holistic 

Modeling 

System) 

Agent  iSolutions 

http :  / /WWW .  agentisolutions .  com/home .  htm 

(an  agent-based  simulation  tool  for  modeling 
the  activities  of  groups  in  different  locations) 

(Brahms,  which  is  strong  on  social  interaction, 
but  weak  on  individual  cognition,  would  make 
a  perfect  match  with  Soar  or  ACT-R) 

No 

(Brahms  includes  a  full  agent- 
(BDI)  and  object-oriented 
language,  you  can  program 
any  measurement  you  want  to 
make  during  a  simulation.) 

Yes 

(Brahms  has  an 
agent-based 
(BDI)  language 
and  a  Java  API. 
Any  human- 
performance 
model  could  be 
interfaced  with 
as  an  agent  in 
Brahms.) 

Yes 

(The  Brahms 
language  allows 
for  initial- 
beliefs  and 
ereation  of 
agents  and 
objeets 
dynamically. 
This  way  you 
can  create 

scenarios  for 
simulation.) 
(The  language 
also  allows  the 
creation  of  new 
beliefs  via 
eonclude 
statements  that 
can  have 
certainty 
factors.  This 
enables 
flexibility  for 
Monte-Carlo 
sims.) 

(Originally  for 
modeling 
business  work 
practice) 
(Brahms  is  a 
domain 
independent 
BDI  language.) 

Yes 

Yes 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

15 

CAST* 

(Collaborative 
Agents  for 
Simulating 
Teamwork) 

The  Permsylvania  State  University 

http://ist.psu.edu/yen/publieations/ijcai01.pdf 

(enables  agents  to  anticipate  potential 
information  needs  among  teammates  and  to 
exchange  information  proactively) 

No,  however  if  other  such 
tools  are  in  use  the  CAST 
framework  provides 
interfaces  through  which  they 
can  be  readily  incorporated. 

Yes 

(a  multi-agent 
architecture) 

(through  plug-in 
modules) 

Yes,  agents 
use  plans  that 
describe  their 
intended 
operation  in 
the  scenario. 

Yes,  CAST 
expects  to 
have  an  API 
provided  by 
the  simulation 
environment 
in  order  to 
interact. 

Yes 

Yes 

16 

STEAM 

(A  Shell  for 
TEAMwork) 

TEAMCORE  Research  Group  at  the 
University  of  Southern  California 

http :  //WWW .  isi .  edu/soar/tambe/ steam/steam,  ht 

ml 

(STEAM  represents  an  integration  of  team 
with  individual  knowledge,  see  Ref  8) 

[GRATE  -  Generic  Rules  and  Agent  model 
Testbed  Environment  (see  Ref  9)  , 

COLLAGEN  -  Collaborative  Agent  (see  Ref 

10)  and  RETSINA  (Reusable  Environment  for 
Task-Structured  Intelligent  Networked  Agents) 
are  examples  of  other  computational  models  of 
teamwork  which  have  been  developed  for 
producing  cooperative  behaviours  among 
intelligent  agents] 

No 

No 

(Based  in  Soar. 
STEAM  is  an 
explicit  model 
of  team  goals 
and  plans  that 
are  shared 
among  team 
members. 

devised  by 
Soar 

developers) 

Yes 

Yes 

Yes 

(“team 

operators”) 

Yes 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

17 

DDD* 

(Distributed 

Dynamic 

Decision- 

Making) 

Aptima 

http :  // w  w  w .  aptima .  com/Projects/Distributed 

Dynamic  Decision  making.html 

(DDD  is  a  synthetic  task 
environment/simulation  engine  that  allows 
researchers  to  prescript  the  actions  of 
opposition  forces.  Though  agents  have  been 
used  in  the  past  with  the  current  version  of  the 
DDD  this  is  challenging.  ) 

No 

(Some  researchers  have  used 
text  based  probes  to 
determine  workload. 
Inferential  techniques  are  also 
a  possibility.) 

See  Ref.  2  for 
integration  of 
the  domain 
independent 
multi-agent 
architecture 
CAST  with  the 
DDD.  CAST- 
DDD  is  an 
architecture  for 
human-agent 
missed  teams  in 
which  agents 
can  replace 
some  or  all  of 
the  players  in  a 
DDD  simulation 
task. 

Yes 

Yes 

Only  for  the 
opposition 
forces 
(OPFOR). 

OPFOR  again 
but  dynamic 
capabilities 
are  limited  to 
spawning  and 
branching. 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

18 

TOD* 

(Team  Optimal 
Design) 

(A  new 
version  of 
TOD  is  under 
development, 
and  hence 
many  answers 
are  related  to 
that  new 
version.) 

Aptima 

http : //www . aptima. com/Projects/Team  Optim 

al  Design.html 

(TOD  is  a  formal,  algorithmic  approach  to 
team  design  that  simultaneously  optimizes 
team  size,  workload  distribution,  and  mission 
tempo  based  on  quantitative  descriptions  of 
task  frequency,  task  workload,  event- task 
flow,  inter-task  communication  requirements, 
and  task  assignment  constraints.) 

Yes 

(In  terms  of  “using”  the 
notion  of  workload. 

TOD  is  optimization  tool, 
and  one  of  the  variables  it 
uses  is  the  workload.) 

No 

(Not  designed 
to  be) 

Yes 

Yes 

No 

Yes 

(Model 
individuals, 
or  sub-teams  - 
groups  of 
individuals.) 

19 

C3TRACE* 

(Command, 
Control,  and 
Communicatio 
n-Techniques 
for  Reliable 
Assessment  of 
Concept 
Execution) 

MA&D  for  ARE  (Ref.  1,  11) 

http :  //www .  dtic.mil/matris/ ddsm/ srch/ddsm02 

21.html 

(An  example  of  integrating  task  and  cognitive 
modeling:  it  takes  the  basic  task  network 
modeling  approach  and  adds  an  information- 
weighted  decision  making  algorithm) 

Yes 

[(Visual,  auditory,  cognitive, 
psychomotor)  VACP 
charmels.  Uses  the 
McCracken/ Aldrich  7-pt 
scale  for  each  channel.] 

No 

(Both  tools  use 
the  Micro  Saint 
Sharp  engine. 

Eiowever, 
IPME  runs  on 
Linux  and  uses 
a  database, 
C3TRACE  runs 
on  Windows 
and  uses  XML) 

Yes 

(Communicati 
ons  can  be 
developed  to 
correspond  to 
any  scenario.) 

No 

[Only  in  the 
(command 
and  control) 

C2  domain.] 

Yes 

Yes 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

20 

KOGSIT 

(Operator 
modelling 
using  cognitive 
architectures) 

Modelling  of  User  Behaviour  in  Dynamic 
Systems  (MoDyS)  Research  Group 

http :  //w  ww .  zmms .  tu- 
berlin.de/ modys/index .  html 

(The  goal  of  this  project  is  to  identify 
requirements  and  propose  modifications  to 
current  cognitive  architectures.) 

No 

Developed  and 
applied 

enhancements  to 
ACT-R/PM  in 
order  to  make 
applied 

modelling  more 
suited  in 
complex  and 
dynamic 
human- 
machine- 
systems  for 
engineering 
purposes 

Yes 

Yes 

No 

No 

21 

Cogitold* 

(an  algorithmic 
model  of  the 
cognitive 
processes 
occurring  in 
the  mind  of 
living 
organisms) 

Institute  of  Computer  Science  at  the  Academy 
of  Sciences  of  the  Czech  Republic. 

http : //WWW . ercim.org/publication/Ercim  New 

No 

No 

Yes 

Yes 

No 

No 

s/enw5  3  /  wiedermarm .  html 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

22 

Archimedes 

Combat 

Modeling 

Platform 

Least  Squares  Software 

http :  //WWW .  leastsquares .  com/papers/mws200 1 
■pdf 

No 

No 

(Agent-based 

modeling) 

Yes 

Yes 

(military 

general) 

Yes 

(Agents  may 
represent 
individuals  or 
units  of  any 
size,  including 
heterogeneous 
collections) 

Yes 

23 

Wildfires 

Fight 

Simulation  for 
Training 

Michigan  Department  of  Natural  Resources 

http :  //WWW .  michigan.  gov/dnr/0 , 1 607 ,7-153- 

10369  36152-1 14699-,00.html 

(The  multi-interactive,  multimedia  program 
uses  audio  and  video  that  involves  the  use  of 
role  players  while  guiding  a  trainee  through  a 
scenario  that  addresses  all  the  strategies, 
tactics  and  safety  issues  he  or  she  might  face  in 
real  fire  incident.) 

No 

No 

Yes 

No 

No 

No 

24 

RESA 

(Research, 
Evaluation, 
and  Systems 
Analysis) 

Space  and  Naval  Warfare  Systems  Command 
(SPAWAR) 

(RESA  is  the  Navy  model  within  the  Joint 
Training  Confederation  (JTC)  of  Models.  It 
provides  Theatre  Level  Simulation 
encompassing  all  Naval  Mission  areas.) 

No 

No,  written  in 
Rational  Eortran 

Yes 

No,  models 
only  Naval 
Activities 
(including 
Air) 

Yes 

Yes 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

25 

EADSIM* 

(Extended  Air 
Defense 
Simulation) 

Teledyne  Brown  Engineering 

http  ://www.eadsim.com/ 

(A  combat  model  -  to  model  the  performance 
and  predict  the  effectiveness  of  ballistic 
missiles,  surface-to-air  missiles,  aircraft,  and 
cruise  missiles  in  a  variety  of  user-developed 
scenarios.) 

Of  the  modeled  entities. 

No 

(Could  be 
compatible  in 
the  future. 
However,  it  is 
likely  it  would 
require  some 
modifications.) 

Yes 

No 

(models  fixed- 
and  rotary¬ 
wing  aircraft, 
tactical 
ballistic 
missiles, 
cruise 
missiles, 
infrared  and 
radar  sensors, 
satellites,  c2 
structures, 
jammers, 
communicatio 
ns  networks 
and  devices, 
and  fire 
support) 

Depends 

(EADSIM 
normally 
models  at  an 
entity  level, 
but  has  some 
ability  to 
aggregate. 

Eor  example, 
a  flight  of 
aircraft  is 
modeled  as 
each 

individual 
aircraft  and 
the 

relationships 
between  them. 

) 

Depends 
(EADSIM 
normally 
models  at  an 
entity  level, 
but  has  some 
ability  to 
aggregate.) 
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No. 

Constructive 

Simulation 

Platform 

Name 

Developer  Organization 

Measure  Workload 

Compatible 
with  IPME 

Scenario 

Flexibility 

Domain 

Independent 

Model  Team 
as  Entity 

Model  Team 
as  Group  of 
Individuals 

26 

JIMM* 

(Joint 
Integrated 
Mission  Model) 
[JIMM  is  a 
language-driven 
model.  If  you 
can  describe  it 
using  the  JIMM 
Conflict 
Language 
(JCL),  you  can 
model  it.  JCL 
is  a  reasonably 
general-purpose 
language  with 
special  support 
for  the  types  of 
issues  (passive 
and  active 
sensing, 
weapons 
engagement, 
communications 
,  human 
decision¬ 
making  in  the 
context  of 
tactics,  etc.) 
relevant  to 
military 
conflict.] 

Air  Force  Agency  for  Modelling  and  Simulation 
(AFAMS) 

http :  // afmsrr .  afams .  af .  mil/index  .cfm?RID  =  MDL 
AF  1000053 

https :  / /WWW  .msrr  .dmso  .mil/share/resourceitem/ 

ViewDetails .  isp?resid = 7707&schema = MSRR 

No 

Probably 

[The  JIMM 
interface  is 
compliant  with 
both  DIS 
(Distributed 
Interactive 
Simulation)  and 
HLA  (High- 
Level 

Architecture)] 

Yes 

Yes 

(JIMM  is 
“general 
purpose”  in 
that  it  is 
highly  flexible 
and  need  not 
be  restricted 
to  military 
applications.) 

Yes 

(JIMM 
executes  the 
scenario  based 
on  the 
underlying 
player 
structures  as 
defined  by  the 
scenario 
programmers, 
not  via 
‘canned’ 
entities.) 

Yes 

(The  specific 
system  types 
modelled  in 
the  scenario 
are  assembled 
into  platform 
types.  One  or 
more  platform 
types  are 
assembled 
into  a  player 
type  where 
tactics  are 
programmed 
to  move  and 
otherwise 
employ 
underlying 
systems) 

F-16 


Team  Modelling:  Seenarios  and  Computational  Models  MAxmamystems 


\i\xnm\sy  stems 


Team  Modelling:  Scenarios  and  Computational  Models 


F-17 


HUM  ANS^ST-fi/S 


Comparison  of  Computational  Models  (Continued) 


No. 

Model 

Specified 

Team 

Tasks 

Model  Specified  Team  Interaction 

Model  HSI  Interventions  of 
interest 

Analyze 

Team’s 

Strategies 

Analyze 

Team’s 

Performance 

Available 
in  Public 
Domain 

Stable 

Real-Time 

Computer 

Generated 

Forces 

1.  IPME* 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

(through  a 
memorandu 
m  of 

agreement) 

Yes 

Yes 

(Dependent  upon 
the  complexity  of 
the  model.) 

2. 

IMPRINT* 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

(Only 

available  to 
US 

Government 
and  their 
contractors) 

Yes 

Yes 

(through 

middlewar) 

3.  ACT-R 

No 

No 

Capable 

No 

No 

Yes 

Yes 

Yes 

4.  MIDAS* 

Yes 

Yes 

(able  to  model  the  interaetion  among  crewmembers) 

Yes 

[E.g.  human  interaetion  with  new 
display  teehniques  for  CEV  (crew 
exploration  vehicle)  ascent.] 

Yes 

Yes 

(Various 
relevant 
outputs, 
including  task 
timelines, 

PERT  charts 
and  situation 
awareness) 

Should  be 
later  this 
year,  after 
the  next 
version  is 
complete 
(couldn't 
easily  use  the 
current 
version). 

Yes 

(But  is  not  yet 
ready  for 
general  public 
use.) 

No 

(Strictly  a  discrete 
time-based 
simulation.) 

\iwaamystems 


Team  Modelling:  Scenarios  and  Computational  Models 


F-19 


HUMANS)  ST- AM/S 


No. 

Model 

Specified 

Team 

Tasks 

Model  Specified  Team  Interaction 

Model  HSI  Interventions  of 
interest 

Analyze 

Team’s 

Strategies 

Analyze 

Team’s 

Performance 

Available 
in  Public 
Domain 

Stable 

Real-Time 

Computer 

Generated 

Forces 

5.  D- 
OMAR* 

Yes 

(Usually 
model  the 
tasks  of 
individuals 
within  a  team 
in  great  detail 
and  then 
examine  team 
behaviours.) 

Yes 

(simulates  interactions  among  multiple  performers) 

(Much  work  has  been  done  on  looking  at  human 
error,  error  sequences  leading  to  incidents  and 
accidents,  and  error  mitigation.) 

Yes 

(Among  others,  examined  and 
evaluated  alternate  flight  deck 
instrument  configurations  using  D- 
OMAR.) 

Yes 

Yes 

(Detailed 
information  on 
individual 
performance 
can  be 
evaluated  to 

assess  team 
performance. 

If  model  a 

team  as  an 
entity,  team 
behaviour  is 
directly 
available  for 
analysis.) 

Yes 

(Open 

source.  Lisp 
version  for 
human 
performance 
modeling; 
Java  version 
for  agent- 
based  system 
development) 

Yes 

(The  Lisp 
version  is  very 
stable.  The  full 
range  of  the 
Java  version 
capabilities  has 
not  been  as 
completely 
exercised.) 

Not  recently,  but 
ever  -  the  very 
first  CGF  tank 
crew  to  operate  at 
Ft.Knox  was  a  D- 
OMAR  model- 
based  four  person 
per  tank,  tank 
platoon  about  20 
years  ago. 

Yes 

6.  SAMPLE 

Yes 

(agent-based  architecture;  model  interagent 
communication) 

Yes 

No 

Yes 

Yes 

Yes 

No 
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No. 

Model 

Specified 

Team 

Tasks 

Model  Specified  Team  Interaction 

Model  HSI  Interventions  of 
interest 

Analyze 

Team’s 

Strategies 

Analyze 

Team’s 

Performance 

Available 
in  Public 
Domain 

Stable 

Real-Time 

Computer 

Generated 

Forces 

7.  GLEAN* 

Yes 

Yes 

Yes 

(Can  represent  different  system 
funetionality  and  interface  design 
deeisions  and  get  performance 
comparisons) 

(A  methodology  that  is  used  to 
identify  the  goals  that  the  operator 
faees  when  interacting  with  a 
technology  or  process  in  system 
design.  GOMS  is  an  attempt  to 
formalize  the  collection  of  the 
activities  that  are  performed  by  the 
operators  in  working  environment. 
This  can  and  often  does  include 
system  interaction  and 
performance.) 

Yes 

(you  can 
compare 
different 

team 

structures 

and 

processes) 

Yes 

No 

(IP  is  owned 
by  University 
of  Michigan, 
but  kernel 
simulator  is 
available  for 
research 
purposes; 
commercial 
rights  are 
reserved. 
Soar 

Technology, 
Inc.  is 

developing  a 
commercial- 
grade 

environment 
for  the  kernel 
simulation 
system.) 

(research 

license 

available) 

Relatively 

(But  research 
on  good 
representation 
of  human 
performance  is 
on  going,  so 
GLEAN  will 
continue  to  be 
upgraded.  Past 
versions  can 
still  be  used) 

Potentially,  but 
not  currently 
used  for  this 
purpose. 

8.  APEX* 

Yes 

Yes 

(Within  the  capabilities  of  Apex's  task  speeifieation 
language,  PDL.) 

Yes 

No 

No 

(Though  there 
is  support  for 
flexible  real¬ 
time  and 
batched  output 
to  analysis 
tools.) 

Yes 

(Through  the 
NASA  Open 
Source 
Agreement.) 

Relatively 

Yes 

(is  one  of  the 
newest  and  has 
not  been 
subjected  to 
the  scrutiny  of 
multiple 
application) 

Yes 

(can  be  integrated 
with  real-time 
systems) 
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HUMANS)5ri-.WS 


No. 

Model 

Specified 

Team 

Tasks 

Model  Specified  Team  Interaction 

Model  HSI  Interventions  of 
interest 

Analyze 

Team’s 

Strategies 

Analyze 

Team’s 

Performance 

Available 
in  Pnbllc 
Domain 

Stable 

Real-Time 

Computer 

Generated 

Forces 

9. 

COGNET 

No 

Yes 

(The  metacognitive  control 

functions,  together  with  the  ability  to  model 
independent  cognitive  agents,  provide 

the  capability,  at  least  in  principle,  to  model 
coordination  among  multiple  team  members) 

Yes 

No 

No 

Yes 

(iGElVP^"  is  a 
commercial 
software  tool 
that  refines 
and  then 

execute 

COGNET) 

Yes 

No 

10.  Soar* 

Yes 

(There's 
nothing  in  the 
architecture 
to  prevent 
this) 

Yes 

(Soar  does  not  include  any  particular  functions  for 
teamwork.  However,  there  have  been  several  models 
of  teamwork  created  in  Soar,  e.g.  Team-soar 

http:  /  /  ieeexplore .  ieee  .org/iel5/3468/2 1 1 90/00983426 . 
pdf?arnumber= 983426 

and  Tac Air-Soar 

http :  /  /  WWW .  soartech .  com/ projects/Tac  Air-Soar  .pdf) 

Yes 

Yes 

(There's 
nothing  in 
the 

architectur 

e  to 

prevent 

this.) 

Yes 

(There's 
nothing  in  the 
architecture  to 
prevent  this.) 

Yes 

(Open 

source) 

Yes 

(The  current 
release  is  8.6, 
and  is  used  by 
researchers 
and 

corporations 
around  the 
world.  Soar 
has  been  under 
development 
for  over  20 
years  and  has 
been  used  in 
major  military 
simulation 
exercise.  Soar 
Technologies 
is  a  40-50 
person 

company  that 
creates  Soar 
models  for 
customers.) 

Yes 

(See 

www.soartech.co 
m  for  multiple 
examples  of  this. 

In  the  STOW-97 
and  STOW-E 
exercises  Soar 
controlled  all  fixed 
and  rotary-wing 
aircraft  fully 
autonomously.) 
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No. 

Model 

Specified 

Team 

Tasks 

Model  Specified  Team  Interaction 

Model  HSI  Interventions  of 
interest 

Analyze 

Team’s 

Strategies 

Analyze 

Team’s 

Performance 

Available 
in  Public 
Domain 

Stable 

Real-Time 

Computer 

Generated 

Forces 

11.  EPIC 

No 

No 

No 

(Primarily  developed  to  model 
HCI) 

No 

No 

Yes 

(source  code 
and 

installation 
instruction 
are  available 
online) 

Yes 

No 

12.  PUMA 

No 

No 

Yes 

(PUMA  is  capable  of  assessing  the 
effeet  on  controller  workload  of 
various  eomputer  assistance  tools.) 

No 

No 

Yes 

(Licensed  to 
third  parties, 
as  a  fully 
supported 
product.) 

Yes 

No 

13.  A-SA 

No 

No 

Yes 

No 

No 

Yes 

Yes 

No 

14.  Brahms* 

Yes 

Yes 

Yes 

Yes 

Yes 

(Brahms  was 
specially 
designed  to 
represent 
aspects  of 
teamwork ) 

Yes 

Yes 

Yes 

15  CAST* 

Yes 

Yes 

No 

No 

Yes  (By 
tracing 
available 
information 
flows  between 

team 

members.) 

Yes 

(Through 

AFOSR.) 

Yes 

No 

(could  be  added) 
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No. 

Model 

Specified 

Team 

Tasks 

Model  Specified  Team  Interaction 

Model  HSI  Interventions  of 
interest 

Analyze 

Team’s 

Strategies 

Analyze 

Team’s 

Performance 

Available 
in  Public 
Domain 

Stable 

Real-Time 

Computer 

Generated 

Forces 

16.  STEAM 

Yes 

Yes 

(has  been  used  to  model  coordination  among  team 
members  in  rotary-wing  companies  and  used  as  the 
underlying  method  for  improving  teamwork  in  the 
Information  Science  Institute  Synthetic  (ISIS)  team 
entered  in  RoboCup  ’97,  an  international  competition 
to  test  multiagent  systems  using  soccer  as  a 
simulation  test  bed.  see  Ref  12). 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

17.  DDD* 

Yes 

Text  based  communications  are  captured  as  are 
all  data  surrounding  actions  requiring  a 
coordinated  action. 

Yes 

(Allow  to  vary  team  structure, 
access  to  information,  and  control 
of  resources) 

Researchers  using  the  DDD 
conduct  analyses  using 
various  statistics  and 
modeling  packages  after 
exporting  the  data  from 
simulation  runs  in  the  DDD. 
Captures  all  data  of  any 
interactions  that  an 
individual,  and  by  default  the 
team,  has  with  the  DDD. 

This  includes  precise  timing 
of  when  the  event  occurred 
and  precisely  where  the  entity 
was  when  the  action  was 
taken.  DDD  provides 
complete  replay  capability. 

Yes 

Usually 

Not  in  the 
dynamie  sense. 

18.  TOD* 

Yes 

Yes 

(helps  to  define  how  the  team  coordinates  its 
execution  of  the  mission’s  tasks) 

Yes 

(helps  to  develop  optimal  team  that 
achieves  the  desired  balance  of 
speed,  individual  workload,  and 
distribution  of  workload  within  the 
team) 

Yes 

(both 

optimizatio 
n  of 

strategies 
and  manual 
setting  is 
available) 

Yes 

No 

(The  new 
version  will 
probably  not 
be  available 
in  public 
domain,  but 
will  be 
distributed  to 
their 
partners) 

Yes 

(current 
version  is 
stable;  new 
version  is 
designed  to  be) 

Yes 

(TOD  has  a  team 
dynamic 
adaptation 
component  that 
can  reconfigure 
the  forces  on-line 
based  on  changes 
in  the  mission.) 
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No. 

Model 

Specified 

Team 

Tasks 

Model  Specified  Team  Interaction 

Model  HSI  Interventions  of 
interest 

Analyze 

Team’s 

Strategies 

Analyze 

Team’s 

Performance 

Available 
in  Public 
Domain 

Stable 

Real-Time 

Computer 

Generated 

Forces 

19. 

C3TRACE* 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

(in  the 
process  of 
developing  a 
policy  for 
foreign 
distribution) 

Yes 

(with 

continual 

upgrading 

and 

enhancement) 

No 

20. 

KOGSIT 

No 

No 

Capable 

No 

No 

Yes 

Unknown 

Yes 

21. 

Cogitoid* 

No 

No 

(A  cogitoid  presents  a  eomputational  model  designed 
with  the  aim  to  model  basic  cognitive  abilities.) 

No 

No 

No 

Yes 

No 

No 

(A  so-called 
'computational 
bacterium' 
driven  by  a 
cogitoid  has 
been  designed, 
which  was  able 
to  learn  to  move 
in  one 
dimension 
towards  a  higher 
concentration  of 
nutrients.) 

22. 

Archimedes 

Yes 

(The 

behavioural 
specification 
is  extremely 
flexible  and 
modular) 

Yes 

(e.g.  “IF  attitude  towards  non-combatants  IS 
friendly  THEN  attitude  towards  militia  IS 
neutral.  END  IF) 

Yes 

Yes 

(roles  of 
discipline, 
cohesion, 
moral  and 
personalit 

y) 

No 

Yes 

Yes 

Yes 

\iwaamystems 


Team  Modelling:  Scenarios  and  Computational  Models 


F-25 
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No. 

Model 

Specified 

Team 

Tasks 

Model  Specified  Team  Interaction 

Model  HSI  Interventions  of 
interest 

Analyze 

Team’s 

Strategies 

Analyze 

Team’s 

Performance 

Available 
in  Public 
Domain 

Stable 

Real-Time 

Computer 

Generated 

Forces 

23.  Forest, 
Mineral, 
and  Fire 
Management 

No 

No 

No 

Yes 

Yes 

Modified 

proprietary 

commercial 

product 

Unknown 

No 

24.  RES  A 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

Yes 

Yes 

25. 

EADSIM* 

Yes 

Yes 

No 

Yes 

Yes 

No 

Yes 

Yes 

26.  JIMM* 

Yes 

Yes 

Yes 

Yes 

Yes 

DoD  and 
DoD 

contractors 

only 

Yes 

Yes 
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